mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

external special remote documentation and example script

Browse source
This commit is contained in:
Joey Hess 2013-12-26 18:14:52 -04:00
parent 38694ed582
commit 0de9135bc0
5 changed files with 171 additions and 71 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

77
doc/design/external_special_remote_protocol.mdwn
View file

@ -1,10 +1,13 @@
See [[todo/support_for_writing_external_special_remotes]] for motivation.
Communication between git-annex and a program implementing an external
special remote uses this protocol.
This is a design for a protocol to be used to communicate between git-annex
and a program implementing an external special remote.
[[!toc]]
## starting the program
The external special remote program has a name like
`git-annex-remote-$bar`. When `git annex initremote foo type=$bar` is run,
`git-annex-remote-$bar`. When
`git annex initremote foo type=external externaltype=$bar` is run,
git-annex finds the appropriate program in PATH.
The program is started by git-annex when it needs to access the special
@ -31,7 +34,7 @@ only sends replies to the requests.
The special remote is responsible for sending the first message, indicating
the version of the protocol it is using.
VERSION 0
VERSION 1
Once it knows the version, git-annex will send a message telling the
special remote to start up.
@ -154,7 +157,7 @@ These messages may be sent by the special remote at any time that it's
in control.
* `VERSION Int`
Supported protocol version. Current version is 0. Must be sent first
Supported protocol version. Current version is 1. Must be sent first
thing at startup, as until it sees this git-annex does not know how to
talk with the special remote program!
* `PROGRESS Int`
@ -200,68 +203,6 @@ remote.
git-annex will not talk to it any further. If the program receives
an ERROR from git-annex, it can exit with its own ERROR.
## Simple shell example
[[!format sh """
#!/bin/sh
set -e
echo VERSION 0
while read line; do
set -- $line
case "$1" in
INITREMOTE)
# XXX do anything necessary to create resources
# used by the remote. Try to be idempotent.
# Use GETCONFIG to get any needed configuration
# settings, and SETCONFIG to set any persistent
# configuration settings.
echo INITREMOTE-SUCCESS
;;
GETCOST)
echo COST-UNKNOWN
;;
PREPARE)
# XXX Use GETCONFIG to get configuration settings,
# and do anything needed to start using the
# special remote here.
echo PREPARE-SUCCESS
;;
TRANSFER)
key="$3"
file="$4"
case "$2" in
STORE)
# XXX upload file here
# XXX when possible, send PROGRESS
echo TRANSFER-SUCCESS STORE "$key"
;;
RETRIEVE)
# XXX download file here
echo TRANSFER-SUCCESS RETRIEVE "$key"
;;
esac
;;
CHECKPRESENT)
key="$2"
echo CHECKPRESENT-UNKNOWN "$key" "not implemented"
;;
REMOVE)
key="$2"
# XXX remove key here
echo REMOVE-SUCCESS "$key"
;;
*)
echo UNKNOWN-REQUEST
;;
esac
done
# XXX anything that needs to be done at shutdown can be done here
"""]]
## TODO
* Communicate when the network connection may have changed, so long-running

20
doc/special_remotes/external.mdwn Normal file
View file

@ -0,0 +1,20 @@
There are three ways to implement a new special remote:
1. Using the [[hook]] special remote to tell git-annex what commands
to run to store and retrieve data. This is the easiest way, and
is great for prototyping.
2. Writing it in Haskell and adding it to git-annex.
3. Writing a program in any language you like that speaks the
[[external_special_remote_protocol]].
This page is all about writing new external special remotes. It's not hard!
* All you need is to make a program with a name like `git-annex-remote-$bar`.
* Install it in PATH.
* When the user runs `git annex initremote foo type=external externaltype=$bar`,
it will use your program.
Here's a simple shell script example, which can easily be adapted
to run whatever commands you need. ([[download|example.sh]])
[[!inline raw=yes pages="special_remotes/external/example.sh"]]

131
doc/special_remotes/external/example.sh vendored Executable file
View file

@ -0,0 +1,131 @@
#!/bin/sh
# git-annex external special remote program
#
# This is basically the same as git-annex's built-in directory special remote.
#
# Install in PATH as git-annex-remote-directorya
#
# Copyright 2013 Joey Hess; licenced under the GNU GPL version 3 or higher.
set -e
# This program speaks a line-based protocol on stdin and stdout.
# When running any commands, their stdout should be redirected to stderr
# (or /dev/null) to avoid messing up the protocol.
runcmd () {
"$@" >&2
}
# Gets a value from the remote's configuration, and stores it in RET
getconfig () {
echo GETCONFIG "$1"
read resp
set -- $resp
case "$1" in
VALUE)
RET="$2"
;;
*)
RET=""
;;
esac
}
# Sets LOC to the location to use to store a key.
mylocation () {
echo HASHDIR "$1"
read resp
set -- $resp
case "$1" in
VALUE)
LOC="$hashdir/$1"
;;
*)
LOC=
;;
esac
}
echo VERSION 1
while read line; do
set -- $line
case "$1" in
INITREMOTE)
# XXX do anything necessary to create resources
# used by the remote. Try to be idempotent.
# Use GETCONFIG to get any needed configuration
# settings, and SETCONFIG to set any persistent
# configuration settings.
getconfig directory
mydirectory="$RET"
if [ -z "$mydirectory" ]; then
echo INITREMOTE-FAILURE "You need to set directory="
else
mkdir -p "$mydirectory"
echo INITREMOTE-SUCCESS
fi
;;
GETCOST)
echo COST-UNKNOWN
;;
PREPARE)
# XXX Use GETCONFIG to get configuration settings,
# and do anything needed to get ready for using the
# special remote here.
getconfig directory
mydirectory="$RET"
;;
TRANSFER)
key="$3"
file="$4"
case "$2" in
STORE)
# XXX upload file here
# XXX when possible, send PROGRESS
calclocation "$key"
mkdir -p "$(dirname "$LOC")"
runcmd cp -v "$file" "$LOC"
echo TRANSFER-SUCCESS STORE "$key"
;;
RETRIEVE)
# XXX download file here
calclocation "$key"
runcmd cp -v "$LOC" "$file"
echo TRANSFER-SUCCESS RETRIEVE "$key"
;;
esac
;;
CHECKPRESENT)
key="$2"
calclocation "$key"
if [ -e "$LOC" ]; then
echo CHECKPRESENT-SUCCESS "$key"
else
if [ -d "$mydirectory" ]; then
echo CHECKPRESENT-FAILURE "$key"
else
# If the directory does not exist,
# the remote is not available.
# (A network remote would similarly
# fail with CHECKPRESENT-UNKNOWN
# if it couldn't be contacted).
echo CHECKPRESENT-UNKNOWN "$key" "this remote is not currently available"
fi
fi
;;
REMOVE)
key="$2"
calclocation "$key"
# Note that it's not a failure to remove a
# key that is not present, so -f is used.
runcmd rm -f "$LOC"
echo REMOVE-SUCCESS "$key"
;;
*)
echo UNKNOWN-REQUEST
;;
esac
done
# XXX anything that needs to be done at shutdown can be done here

12
doc/special_remotes/hook.mdwn
View file

@ -1,9 +1,12 @@
This special remote type lets you store content in a remote of your own
devising.
devising, configured via some simple hooks.
It's not recommended to use this remote type when another like [[rsync]]
or [[directory]] will do. If your hooks are not carefully written, data
could be lost.
could be lost.
If you're building a special remote for others to use,
instead consider building an [[external_special_remote|external]].
## example
@ -68,6 +71,9 @@ The settings to use in git config for the hook commands are as follows:
## combined hook program
This interface is deprecated -- it's better, and not much harder,
to write an [[external_special_remote|external]]!
Rather than setting all of the above hooks, you can write a single
program that handles everything, and set a single hook to make it be used.
@ -75,7 +81,7 @@ program that handles everything, and set a single hook to make it be used.
# git annex initremote mydemorepo type=hook hooktype=demo encryption=none
The program just needs to look at the `ANNEX_ACTION` environment variable
to see what it's being asked to do For example:
to see what it's being asked to do. For example:
[[!format sh """
#!/bin/sh

2
doc/todo/support_for_writing_external_special_remotes.mdwn
View file

@ -23,3 +23,5 @@ should look for `git-annex-remote-$bar` in PATH if that's not a built-in
special remote name.
--[[Joey]]
[[done]]