218 lines
8.8 KiB
Markdown
218 lines
8.8 KiB
Markdown
Communication between git-annex and a program implementing an external
|
|
special remote uses this protocol.
|
|
|
|
[[!toc]]
|
|
|
|
## starting the program
|
|
|
|
The external special remote program has a name like
|
|
`git-annex-remote-$bar`. When
|
|
`git annex initremote foo type=external externaltype=$bar` is run,
|
|
git-annex finds the appropriate program in PATH.
|
|
|
|
The program is started by git-annex when it needs to access the special
|
|
remote, and may be left running for a long period of time. This allows
|
|
it to perform expensive setup tasks, etc. Note that git-annex may choose to
|
|
start multiple instances of the program (eg, when multiple git-annex
|
|
commands are run concurrently in a repository).
|
|
|
|
## protocol overview
|
|
|
|
Communication is via stdin and stdout. Therefore, the external special
|
|
remote must avoid doing any prompting, or outputting anything like eg,
|
|
progress to stdout. (Such stuff can be sent to stderr instead.)
|
|
|
|
The protocol is line based. Messages are sent in either direction, from
|
|
git-annex to the special remote, and from the special remote to git-annex.
|
|
|
|
In order to avoid confusing interactions, one or the other has control
|
|
at any given time, and is responsible for sending requests, while the other
|
|
only sends replies to the requests.
|
|
|
|
Each protocol line starts with a command, which is followed by the
|
|
command's parameters (a fixed number per command), each separated by a
|
|
single space. The last parameter may contain spaces. Parameters may be
|
|
empty, but the separating spaces are still required in that case.
|
|
|
|
## example session
|
|
|
|
The special remote is responsible for sending the first message, indicating
|
|
the version of the protocol it is using.
|
|
|
|
VERSION 1
|
|
|
|
Once it knows the version, git-annex will generally
|
|
send a message telling the special remote to start up.
|
|
(Or it might send a INITREMOTE, so don't hardcode this order.)
|
|
|
|
PREPARE
|
|
|
|
The special remote can now ask git-annex for its configuration, as needed,
|
|
and check that it's valid. git-annex responds with the configuration values
|
|
|
|
GETCONFIG directory
|
|
VALUE /media/usbdrive/repo
|
|
GETCONFIG automount
|
|
VALUE true
|
|
|
|
Once the special remote is satisfied with its configuration and is
|
|
ready to go, it tells git-annex.
|
|
|
|
PREPARE-SUCCESS
|
|
|
|
Now git-annex will tell the special remote what to do. Let's suppose
|
|
it wants to store a key.
|
|
|
|
TRANSFER STORE somekey tmpfile
|
|
|
|
The special remote can continue sending messages to git-annex during this
|
|
transfer. It will typically send progress messages, indicating how many
|
|
bytes have been sent:
|
|
|
|
PROGRESS 10240
|
|
PROGRESS 20480
|
|
|
|
Once the key has been stored, the special remote tells git-annex the result:
|
|
|
|
TRANSFER-SUCCESS STORE somekey
|
|
|
|
Once git-annex is done with the special remote, it will close its stdin.
|
|
The special remote program can then exit.
|
|
|
|
## git-annex request messages
|
|
|
|
These are messages git-annex sends to the special remote program.
|
|
None of these messages require an immediate reply. The special
|
|
remote can send any messages it likes while handling the requests.
|
|
|
|
Once the special remote has finished performing the request, it should
|
|
send one of the corresponding replies listed in the next section.
|
|
|
|
The following requests *must* all be supported by the special remote.
|
|
|
|
* `INITREMOTE`
|
|
Request that the remote initialize itself. This is where any one-time
|
|
setup tasks can be done, for example creating an Amazon S3 bucket.
|
|
Note: This may be run repeatedly over time, as a remote is initialized in
|
|
different repositories, or as the configuration of a remote is changed.
|
|
So any one-time setup tasks should be done idempotently.
|
|
* `PREPARE`
|
|
Tells the special remote it's time to prepare itself to be used.
|
|
Only INITREMOTE can come before this.
|
|
* `TRANSFER STORE|RETRIEVE Key File`
|
|
Requests the transfer of a key. For Send, the File is the file to upload;
|
|
for Receive the File is where to store the download.
|
|
Note that the File should not influence the filename used on the remote.
|
|
The filename will not contain any whitespace.
|
|
Multiple transfers might be requested by git-annex, but it's fine for the
|
|
program to serialize them and only do one at a time.
|
|
* `CHECKPRESENT Key`
|
|
Requests the remote check if a key is present in it.
|
|
* `REMOVE Key`
|
|
Requests the remote remove a key's contents.
|
|
|
|
The following requests can optionally be supported. If not handled,
|
|
replying with `UNSUPPORTED-REQUEST` is acceptable.
|
|
|
|
* `GETCOST`
|
|
Requests the remote return a use cost. Higher costs are more expensive.
|
|
(See Config/Cost.hs for some standard costs.)
|
|
|
|
More optional requests may be added, without changing the protocol version,
|
|
so if an unknown request is seen, reply with `UNSUPPORTED-REQUEST`.
|
|
|
|
## special remote replies
|
|
|
|
These should be sent only in response to the git-annex request messages.
|
|
They do not have to be sent immediately after the request; the special
|
|
remote can send its own requests (listed in the next section below)
|
|
while it's handling a request.
|
|
|
|
* `PREPARE-SUCCESS`
|
|
Sent as a response to PREPARE once the special remote is ready for use.
|
|
* `TRANSFER-SUCCESS STORE|RETRIEVE Key`
|
|
Indicates the transfer completed successfully.
|
|
* `TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg`
|
|
Indicates the transfer failed.
|
|
* `CHECKPRESENT-SUCCESS Key`
|
|
Indicates that a key has been positively verified to be present in the
|
|
remote.
|
|
* `CHECKPRESENT-FAILURE Key`
|
|
Indicates that a key has been positively verified to not be present in the
|
|
remote.
|
|
* `CHECKPRESENT-UNKNOWN Key ErrorMsg`
|
|
Indicates that it is not currently possible to verify if the key is
|
|
present in the remote. (Perhaps the remote cannot be contacted.)
|
|
* `REMOVE-SUCCESS Key`
|
|
Indicates the key has been removed from the remote. May be returned if
|
|
the remote didn't have the key at the point removal was requested.
|
|
* `REMOVE-FAILURE Key ErrorMsg`
|
|
Indicates that the key was unable to be removed from the remote.
|
|
* `COST Int`
|
|
Indicates the cost of the remote.
|
|
* `INITREMOTE-SUCCESS`
|
|
Indicates the INITREMOTE succeeded and the remote is ready to use.
|
|
* `INITREMOTE-FAILURE ErrorMsg`
|
|
Indicates that INITREMOTE failed.
|
|
* `UNSUPPORTED-REQUEST`
|
|
Indicates that the special remote does not know how to handle a request.
|
|
|
|
## special remote messages
|
|
|
|
These messages may be sent by the special remote at any time that it's
|
|
in control.
|
|
|
|
* `VERSION Int`
|
|
Supported protocol version. Current version is 1. Must be sent first
|
|
thing at startup, as until it sees this git-annex does not know how to
|
|
talk with the special remote program!
|
|
* `PROGRESS Int`
|
|
Indicates the current progress of the transfer (in bytes). May be repeated
|
|
any number of times during the transfer process. This is highly recommended
|
|
for STORE. (It is optional but good for RETRIEVE.)
|
|
(git-annex does not send a reply to this message.)
|
|
* `DIRHASH Key`
|
|
Gets a two level hash associated with a Key. Something like "abc/def".
|
|
This is always the same for any given Key, so can be used for eg,
|
|
creating hash directory structures to store Keys in.
|
|
(git-annex replies with VALUE followed by the value.)
|
|
* `SETCONFIG Setting`
|
|
Sets one of the special remote's configuration settings.
|
|
Normally this is sent during INITREMOTE, which allows these settings
|
|
to be stored in the git-annex branch, so will be available if the same
|
|
special remote is used elsewhere. (If sent after INITREMOTE, the changed
|
|
configuration will only be available while the remote is running.)
|
|
* `GETCONFIG Setting`
|
|
Gets one of the special remote's configuration settings, which can have
|
|
been passed by the user when running `git annex initremote`, or
|
|
can have been set by a previous SETCONFIG. Can be run at any time.
|
|
(git-annex replies with VALUE followed by the value. If the setting is
|
|
not set, the value will be empty.)
|
|
|
|
## general messages
|
|
|
|
These messages can be sent at any time by either git-annex or the special
|
|
remote.
|
|
|
|
* `ERROR ErrorMsg`
|
|
Generic error. Can be sent at any time if things get too messed up
|
|
to continue. When possible, use a more specific reply from the list above.
|
|
The special remote program should exit after sending this, as
|
|
git-annex will not talk to it any further. If the program receives
|
|
an ERROR from git-annex, it can exit with its own ERROR.
|
|
|
|
## TODO
|
|
|
|
* When storing encrypted files stream the file up/down the pipe, rather
|
|
than using a temp file. Will probably involve less space and disk IO,
|
|
and makes the progress display better, since the encryption can happen
|
|
concurrently with the transfer. Also, no need to use PROGRESS in this
|
|
scenario, since git-annex can see how much data it has sent/received from
|
|
the remote. However, \n and probably \0 need to be escaped somehow in the
|
|
file data, which adds complication.
|
|
* uuid discovery during INITREMOTE.
|
|
* uuid verification during PREPARE (so, for example, it can check if a
|
|
removable drive repo has the expected uuid)
|
|
* Support for splitting files into chunks.
|
|
* Support for getting and setting the list of urls that can be associated
|
|
with a key.
|