experimental ipfs special remote, with addurl support
This commit is contained in:
Joey Hess
parent
e7b8eccef0
commit
ea74b1d568
3 changed files with 212 additions and 0 deletions
|
|
@ -42,6 +42,7 @@ for using git-annex with various services:
|
|||
* [chef-vault](https://github.com/3ofcoins/knife-annex/)
|
||||
* [hubiC](https://github.com/Schnouki/git-annex-remote-hubic)
|
||||
* [pCloud](https://github.com/tochev/git-annex-remote-pcloud)
|
||||
* [[ipfs]]
|
||||
|
||||
Want to add support for something else? [[Write your own!|external]]
|
||||
|
||||
|
|
|
|||
125
doc/special_remotes/external/git-annex-remote-ipfs
vendored
Executable file
125
doc/special_remotes/external/git-annex-remote-ipfs
vendored
Executable file
|
|
@ -0,0 +1,125 @@
|
|||
#!/bin/sh
|
||||
# This is a git-annex external special remote program,
|
||||
# which adds experimental ipfs support to git-annex.
|
||||
#
|
||||
# Install in PATH as git-annex-remote-ipfs
|
||||
#
|
||||
# Copyright 2015 Joey Hess; licenced under the GNU GPL version 3 or higher.
|
||||
|
||||
set -e
|
||||
|
||||
# use ipfs: as a prefix to indicate when an "url" is really stored in ipfs
|
||||
isipfsurl () {
|
||||
echo "$1" | egrep -q "^ipfs:"
|
||||
}
|
||||
|
||||
# convert an ipfs: url to an address that the ipfs client understands
|
||||
urltoaddress () {
|
||||
echo "$1" | sed -e 's/^ipfs://'
|
||||
}
|
||||
|
||||
addresstourl () {
|
||||
echo "ipfs:$1"
|
||||
}
|
||||
|
||||
# Gets a VALUE response and stores it in $RET
|
||||
getvalue () {
|
||||
read resp
|
||||
# Tricky POSIX shell code to split first word of the resp,
|
||||
# preserving all other whitespace
|
||||
case "${resp%% *}" in
|
||||
VALUE)
|
||||
RET="$(echo "$resp" | sed 's/^VALUE \?//')"
|
||||
;;
|
||||
*)
|
||||
RET=""
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Get a list of all known ipfs addresses for a key,
|
||||
# storing it in a temp file.
|
||||
getaddrs () {
|
||||
key="$1"
|
||||
tmp="$2"
|
||||
|
||||
echo GETURLS "$key"
|
||||
getvalue
|
||||
while [ -n "$RET" ]; do
|
||||
if isipfsurl "$RET"; then
|
||||
echo "$RET" >> "$tmp"
|
||||
fi
|
||||
getvalue
|
||||
done
|
||||
}
|
||||
|
||||
# This has to come first, to get the protocol started.
|
||||
echo VERSION 1
|
||||
|
||||
while read line; do
|
||||
set -- $line
|
||||
case "$1" in
|
||||
INITREMOTE)
|
||||
echo INITREMOTE-SUCCESS
|
||||
;;
|
||||
PREPARE)
|
||||
echo PREPARE-SUCCESS
|
||||
;;
|
||||
CLAIMURL)
|
||||
url="$2"
|
||||
if isipfsurl "$url"; then
|
||||
echo CLAIMURL-SUCCESS
|
||||
else
|
||||
echo CLAIMURL-FAILURE
|
||||
fi
|
||||
;;
|
||||
CHECKURL)
|
||||
url="$2"
|
||||
# TODO if size of file can be quickly determined
|
||||
# (without downloading it) return the size
|
||||
# instead of UNKNOWN
|
||||
echo CHECKURL-CONTENTS UNKNOWN "$(urltoaddress "$url")"
|
||||
;;
|
||||
TRANSFER)
|
||||
key="$3"
|
||||
file="$4"
|
||||
case "$2" in
|
||||
STORE)
|
||||
addr=$(ipfs add -q "$file" </dev/null) || true
|
||||
if [ -z "$addr" ]; then
|
||||
echo TRANSFER-FAILURE STORE "$key" "ipfs add failed"
|
||||
else
|
||||
echo "SETURLPRESENT" "$key" "$(addresstourl "$addr")"
|
||||
echo TRANSFER-SUCCESS STORE "$key"
|
||||
fi
|
||||
;;
|
||||
RETRIEVE)
|
||||
addrtmp=$(mktemp)
|
||||
getaddrs "$key" "$addrtmp"
|
||||
addr="$(urltoaddress "$(head "$addrtmp")")" || true
|
||||
rm -f "$addrtmp"
|
||||
if [ -z "$addr" ]; then
|
||||
echo TRANSFER-FAILURE RETRIEVE "$key" "no known ipfs address for this key"
|
||||
else
|
||||
if ! ipfs get --output="$file" "$addr" >&2 </dev/null; then
|
||||
echo TRANSFER-FAILURE RETRIEVE "$key" "failed downloading ipfs $addr"
|
||||
else
|
||||
echo TRANSFER-SUCCESS RETRIEVE "$key"
|
||||
fi
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
CHECKPRESENT)
|
||||
key="$2"
|
||||
echo CHECKPRESENT-FAILURE "$key"
|
||||
;;
|
||||
REMOVE)
|
||||
key="$2"
|
||||
echo REMOVE-FAILURE "$key" "cannot remove content from ipfs (instead, run ipfs gc to clear your local ipfs cache)"
|
||||
;;
|
||||
*)
|
||||
echo UNSUPPORTED-REQUEST
|
||||
;;
|
||||
esac
|
||||
done
|
||||
86
doc/special_remotes/ipfs.mdwn
Normal file
86
doc/special_remotes/ipfs.mdwn
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
This special remote stores file contents in [ipfs](http://ipfs.io/).
|
||||
|
||||
Warning: As this page is being written, ipfs is still considered alpha
|
||||
quality code, not suitable for production use. Still, it's fun to play
|
||||
with, has some nice features and great potential, and git-annex can
|
||||
keep your data safe while you're using ipfs.
|
||||
|
||||
## prerequisites
|
||||
|
||||
* git-annex version 5.20141219 or newer, which has [[external]] special remote
|
||||
support.
|
||||
* Install [[external/git-annex-remote-ipfs]] somewhere in PATH
|
||||
and `chmod +x` the script.
|
||||
* Install [go-ipfs](https://github.com/jbenet/go-ipfs) somewhere in PATH.
|
||||
* Run `ipfs init` and start the `ipfs daemon`
|
||||
|
||||
(Note that this special remote does not use ipfs's FUSE support; it
|
||||
communicates with ipfs using the `ipfs` command-line utility.)
|
||||
|
||||
## configuration
|
||||
|
||||
These parameters can be passed to `git annex initremote` to configure the
|
||||
remote:
|
||||
|
||||
* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
|
||||
See [[encryption]]. Note that this is git-annex's encryption, not ipfs's
|
||||
encryption.
|
||||
|
||||
* `keyid` - Specifies the gpg key to use for [[encryption]].
|
||||
|
||||
Setup example:
|
||||
|
||||
# git annex initremote ipfs type=external externaltype=ipfs encryption=none
|
||||
|
||||
## content distribution
|
||||
|
||||
After `git annex copy --to ipfs`, a file will typically only have
|
||||
been copied to your computer's local ipfs object store. It will not reach
|
||||
other ipfs nodes on the network until they request the content.
|
||||
|
||||
If you set up a clone of your repository on another computer, and install
|
||||
ipfs and enable the ipfs remote there, you can proceed with using it to get
|
||||
files that have been stored in ipfs:
|
||||
|
||||
# git annex sync
|
||||
# git annex enableremote ipfs
|
||||
# git annex copy --from ipfs
|
||||
|
||||
# content removal
|
||||
|
||||
Removing content from ipfs requires all nodes that have a copy to decide to
|
||||
delete it. This is not something git-annex can arrange to happen, or
|
||||
reliably tell has happened, so `git annex drop --from ipfs` will always fail.
|
||||
|
||||
## using ipfs addresses
|
||||
|
||||
Once a file has been copied to ipfs, you can use `git annex whereis`
|
||||
to look up the ipfs address of the file:
|
||||
|
||||
# git annex whereis somefile
|
||||
whereis somefile
|
||||
ed1c811d-fe42-4436-aa75-56566c990aa8 -- ipfs
|
||||
|
||||
ipfs: QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W
|
||||
|
||||
In the example above, the ipfs address for the file is
|
||||
`QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W`. You can give this
|
||||
address to any other ipfs user and they can use it to download the file!
|
||||
|
||||
You can also use ipfs addresses with `git annex addurl`. For example:
|
||||
|
||||
# git annex addurl ipfs:QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W --file somefile
|
||||
|
||||
That's a real file; try it!
|
||||
|
||||
## future directions
|
||||
|
||||
While perhaps useful, this is just a proof of concept. It's particularly
|
||||
lacking in that it doesn't integrate well git-annex's [[location_tracking]]
|
||||
with ipfs.
|
||||
|
||||
Tracking which ipfs nodes have a copy of an annexed object
|
||||
would make this special remote work better. In particular, git-annex does
|
||||
not currently trust ipfs to contain a copy of an object, since it has no
|
||||
way of keeping track of which which ipfs nodes might contain it. So, eg,
|
||||
`git annex drop` will refuse to trust ipfs.
|
||||
Loading…
Reference in a new issue