experimental ipfs special remote, with addurl support
This commit is contained in:
parent
e7b8eccef0
commit
ea74b1d568
3 changed files with 212 additions and 0 deletions
|
@ -42,6 +42,7 @@ for using git-annex with various services:
|
||||||
* [chef-vault](https://github.com/3ofcoins/knife-annex/)
|
* [chef-vault](https://github.com/3ofcoins/knife-annex/)
|
||||||
* [hubiC](https://github.com/Schnouki/git-annex-remote-hubic)
|
* [hubiC](https://github.com/Schnouki/git-annex-remote-hubic)
|
||||||
* [pCloud](https://github.com/tochev/git-annex-remote-pcloud)
|
* [pCloud](https://github.com/tochev/git-annex-remote-pcloud)
|
||||||
|
* [[ipfs]]
|
||||||
|
|
||||||
Want to add support for something else? [[Write your own!|external]]
|
Want to add support for something else? [[Write your own!|external]]
|
||||||
|
|
||||||
|
|
125
doc/special_remotes/external/git-annex-remote-ipfs
vendored
Executable file
125
doc/special_remotes/external/git-annex-remote-ipfs
vendored
Executable file
|
@ -0,0 +1,125 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# This is a git-annex external special remote program,
|
||||||
|
# which adds experimental ipfs support to git-annex.
|
||||||
|
#
|
||||||
|
# Install in PATH as git-annex-remote-ipfs
|
||||||
|
#
|
||||||
|
# Copyright 2015 Joey Hess; licenced under the GNU GPL version 3 or higher.
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# use ipfs: as a prefix to indicate when an "url" is really stored in ipfs
|
||||||
|
isipfsurl () {
|
||||||
|
echo "$1" | egrep -q "^ipfs:"
|
||||||
|
}
|
||||||
|
|
||||||
|
# convert an ipfs: url to an address that the ipfs client understands
|
||||||
|
urltoaddress () {
|
||||||
|
echo "$1" | sed -e 's/^ipfs://'
|
||||||
|
}
|
||||||
|
|
||||||
|
addresstourl () {
|
||||||
|
echo "ipfs:$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Gets a VALUE response and stores it in $RET
|
||||||
|
getvalue () {
|
||||||
|
read resp
|
||||||
|
# Tricky POSIX shell code to split first word of the resp,
|
||||||
|
# preserving all other whitespace
|
||||||
|
case "${resp%% *}" in
|
||||||
|
VALUE)
|
||||||
|
RET="$(echo "$resp" | sed 's/^VALUE \?//')"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
RET=""
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get a list of all known ipfs addresses for a key,
|
||||||
|
# storing it in a temp file.
|
||||||
|
getaddrs () {
|
||||||
|
key="$1"
|
||||||
|
tmp="$2"
|
||||||
|
|
||||||
|
echo GETURLS "$key"
|
||||||
|
getvalue
|
||||||
|
while [ -n "$RET" ]; do
|
||||||
|
if isipfsurl "$RET"; then
|
||||||
|
echo "$RET" >> "$tmp"
|
||||||
|
fi
|
||||||
|
getvalue
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# This has to come first, to get the protocol started.
|
||||||
|
echo VERSION 1
|
||||||
|
|
||||||
|
while read line; do
|
||||||
|
set -- $line
|
||||||
|
case "$1" in
|
||||||
|
INITREMOTE)
|
||||||
|
echo INITREMOTE-SUCCESS
|
||||||
|
;;
|
||||||
|
PREPARE)
|
||||||
|
echo PREPARE-SUCCESS
|
||||||
|
;;
|
||||||
|
CLAIMURL)
|
||||||
|
url="$2"
|
||||||
|
if isipfsurl "$url"; then
|
||||||
|
echo CLAIMURL-SUCCESS
|
||||||
|
else
|
||||||
|
echo CLAIMURL-FAILURE
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
CHECKURL)
|
||||||
|
url="$2"
|
||||||
|
# TODO if size of file can be quickly determined
|
||||||
|
# (without downloading it) return the size
|
||||||
|
# instead of UNKNOWN
|
||||||
|
echo CHECKURL-CONTENTS UNKNOWN "$(urltoaddress "$url")"
|
||||||
|
;;
|
||||||
|
TRANSFER)
|
||||||
|
key="$3"
|
||||||
|
file="$4"
|
||||||
|
case "$2" in
|
||||||
|
STORE)
|
||||||
|
addr=$(ipfs add -q "$file" </dev/null) || true
|
||||||
|
if [ -z "$addr" ]; then
|
||||||
|
echo TRANSFER-FAILURE STORE "$key" "ipfs add failed"
|
||||||
|
else
|
||||||
|
echo "SETURLPRESENT" "$key" "$(addresstourl "$addr")"
|
||||||
|
echo TRANSFER-SUCCESS STORE "$key"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
RETRIEVE)
|
||||||
|
addrtmp=$(mktemp)
|
||||||
|
getaddrs "$key" "$addrtmp"
|
||||||
|
addr="$(urltoaddress "$(head "$addrtmp")")" || true
|
||||||
|
rm -f "$addrtmp"
|
||||||
|
if [ -z "$addr" ]; then
|
||||||
|
echo TRANSFER-FAILURE RETRIEVE "$key" "no known ipfs address for this key"
|
||||||
|
else
|
||||||
|
if ! ipfs get --output="$file" "$addr" >&2 </dev/null; then
|
||||||
|
echo TRANSFER-FAILURE RETRIEVE "$key" "failed downloading ipfs $addr"
|
||||||
|
else
|
||||||
|
echo TRANSFER-SUCCESS RETRIEVE "$key"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
;;
|
||||||
|
CHECKPRESENT)
|
||||||
|
key="$2"
|
||||||
|
echo CHECKPRESENT-FAILURE "$key"
|
||||||
|
;;
|
||||||
|
REMOVE)
|
||||||
|
key="$2"
|
||||||
|
echo REMOVE-FAILURE "$key" "cannot remove content from ipfs (instead, run ipfs gc to clear your local ipfs cache)"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo UNSUPPORTED-REQUEST
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
86
doc/special_remotes/ipfs.mdwn
Normal file
86
doc/special_remotes/ipfs.mdwn
Normal file
|
@ -0,0 +1,86 @@
|
||||||
|
This special remote stores file contents in [ipfs](http://ipfs.io/).
|
||||||
|
|
||||||
|
Warning: As this page is being written, ipfs is still considered alpha
|
||||||
|
quality code, not suitable for production use. Still, it's fun to play
|
||||||
|
with, has some nice features and great potential, and git-annex can
|
||||||
|
keep your data safe while you're using ipfs.
|
||||||
|
|
||||||
|
## prerequisites
|
||||||
|
|
||||||
|
* git-annex version 5.20141219 or newer, which has [[external]] special remote
|
||||||
|
support.
|
||||||
|
* Install [[external/git-annex-remote-ipfs]] somewhere in PATH
|
||||||
|
and `chmod +x` the script.
|
||||||
|
* Install [go-ipfs](https://github.com/jbenet/go-ipfs) somewhere in PATH.
|
||||||
|
* Run `ipfs init` and start the `ipfs daemon`
|
||||||
|
|
||||||
|
(Note that this special remote does not use ipfs's FUSE support; it
|
||||||
|
communicates with ipfs using the `ipfs` command-line utility.)
|
||||||
|
|
||||||
|
## configuration
|
||||||
|
|
||||||
|
These parameters can be passed to `git annex initremote` to configure the
|
||||||
|
remote:
|
||||||
|
|
||||||
|
* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
|
||||||
|
See [[encryption]]. Note that this is git-annex's encryption, not ipfs's
|
||||||
|
encryption.
|
||||||
|
|
||||||
|
* `keyid` - Specifies the gpg key to use for [[encryption]].
|
||||||
|
|
||||||
|
Setup example:
|
||||||
|
|
||||||
|
# git annex initremote ipfs type=external externaltype=ipfs encryption=none
|
||||||
|
|
||||||
|
## content distribution
|
||||||
|
|
||||||
|
After `git annex copy --to ipfs`, a file will typically only have
|
||||||
|
been copied to your computer's local ipfs object store. It will not reach
|
||||||
|
other ipfs nodes on the network until they request the content.
|
||||||
|
|
||||||
|
If you set up a clone of your repository on another computer, and install
|
||||||
|
ipfs and enable the ipfs remote there, you can proceed with using it to get
|
||||||
|
files that have been stored in ipfs:
|
||||||
|
|
||||||
|
# git annex sync
|
||||||
|
# git annex enableremote ipfs
|
||||||
|
# git annex copy --from ipfs
|
||||||
|
|
||||||
|
# content removal
|
||||||
|
|
||||||
|
Removing content from ipfs requires all nodes that have a copy to decide to
|
||||||
|
delete it. This is not something git-annex can arrange to happen, or
|
||||||
|
reliably tell has happened, so `git annex drop --from ipfs` will always fail.
|
||||||
|
|
||||||
|
## using ipfs addresses
|
||||||
|
|
||||||
|
Once a file has been copied to ipfs, you can use `git annex whereis`
|
||||||
|
to look up the ipfs address of the file:
|
||||||
|
|
||||||
|
# git annex whereis somefile
|
||||||
|
whereis somefile
|
||||||
|
ed1c811d-fe42-4436-aa75-56566c990aa8 -- ipfs
|
||||||
|
|
||||||
|
ipfs: QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W
|
||||||
|
|
||||||
|
In the example above, the ipfs address for the file is
|
||||||
|
`QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W`. You can give this
|
||||||
|
address to any other ipfs user and they can use it to download the file!
|
||||||
|
|
||||||
|
You can also use ipfs addresses with `git annex addurl`. For example:
|
||||||
|
|
||||||
|
# git annex addurl ipfs:QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W --file somefile
|
||||||
|
|
||||||
|
That's a real file; try it!
|
||||||
|
|
||||||
|
## future directions
|
||||||
|
|
||||||
|
While perhaps useful, this is just a proof of concept. It's particularly
|
||||||
|
lacking in that it doesn't integrate well git-annex's [[location_tracking]]
|
||||||
|
with ipfs.
|
||||||
|
|
||||||
|
Tracking which ipfs nodes have a copy of an annexed object
|
||||||
|
would make this special remote work better. In particular, git-annex does
|
||||||
|
not currently trust ipfs to contain a copy of an object, since it has no
|
||||||
|
way of keeping track of which which ipfs nodes might contain it. So, eg,
|
||||||
|
`git annex drop` will refuse to trust ipfs.
|
Loading…
Reference in a new issue