Communication between git-annex and a program implementing an external
[[backend|backends]] uses this protocol.

[[!toc ]]

## starting the program

The external backend program has a name like `git-annex-backend-XFOO`.
When git-annex is configured to use a backend starting with "X", 
or encounters a key in a repository starting with "X", it
looks for the corresponding external backend program in PATH.

The program is started by git-annex when it needs to use it, and may be
left running for a long period of time. Note that git-annex may choose to
run multiple instances of the program.

## protocol overview

Communication is via stdin and stdout. While stderr is connected to the
console and so visible to the user, the program should avoid using it
except for in the most exceptional circumstances.

The protocol is line based. git-annex sends a request, and the program
responds with a reply.

Each protocol line starts with a command, which is followed by the
command's parameters (a fixed number per command), each separated by a
single space. The last parameter may contain spaces. Parameters may be
empty, but the separating spaces are still required in that case.

## example session

git-annex always starts by sending a message asking the program what protocol
version it uses.

	GETVERSION

The program responds.

	VERSION 1

git-annex will next query the program about the properties of the keys it
uses (`CANVERIFY`, `ISSTABLE`, `ISCRYPTOGRAPHICALLYSECURE`), and the program will
respond to each query.

Then git-annex may ask the program to generate a key.

	GENKEY somefile

The program will respond with the key it generated, but if it needs to do
an expensive operation, such as hashing the file, it can first send
progress messages, indicating the position in the file it has processed.

	PROGRESS 1024
	PROGRESS 2048
	GENKEY-SUCCESS XFOO-s2048--dbd009

git-annex can also ask the program to verify if the content of a file
matches a key.

	VERIFYKEYCONTENT XFOO-s2048--dbd009 somefile

Again the program can send progress messages as it works, finishing
with the result of the verification.

	PROGRESS 1024
	PROGRESS 2048
	VERIFYKEYCONTENT-SUCCESS

## startup messages and replies

These messages are sent to the program soon after starting it, and it should
reply with one of the listed replies.

* `GETVERSION`  
  Always the first message sent.  
  Currently the only version of this protocol is version 1.
  * `VERSION 1`  
* `CANVERIFY`  
  Asks if the program can verify the content of files match a key it generated.
  The verification does not need to be cryptographically secure, but should
  catch data corruption.
  * `CANVERIFY-YES`
  * `CANVERIFY-NO`
* `ISSTABLE`  
  Asks the program if a key it has generated will always have the same
  content. The answer to this is almost always yes; URL keys are an example
  of a type of key that may have different content at different times.
  * `ISSTABLE-YES`
  * `ISSTABLE-NO`
* `ISCRYPTOGRAPHICALLYSECURE`  
  Asks the program if keys it generates are verified using a cryptographically
  secure hash. Note that sha1 is *not* a cryptographically secure hash any
  longer. A program can change its answer to this question as the state of the
  art advances, and should aim to stay ahead of the state of the art by a
  reasonable amount of time.
  * `ISCRYPTOGRAPHICALLYSECURE-YES`
  * `ISCRYPTOGRAPHICALLYSECURE-NO`

## main messages and replies

This is where work happens.

* `GENKEY Contentfile`  
  The program should examine the ContentFile and from it generate a
  key. While it is doing this, it can send any number of `PROGRESS`
  messages indication the position in the file that it's gotten to.
  * `GENKEY-SUCCESS Key`
  * `GENKEY-FAILURE ErrorMsg`
* `VERIFYKEYCONTENT Key ContentFile`  
  The program should examine the ContentFile and verify that it has the
  content it would expect for the Key. While it is doing this, it can
  send any number of `PROGRESS` messages indication the position in the
  file that it's gotten to. (If the program earlier sent `CANVERIFY-NO`,
  it will not be asked to do this.)
  * `VERIFYKEYCONTENT-SUCCESS`
  * `VERIFYKEYCONTENT-FAILURE`

## general messages

These messages can be sent at any time by either git-annex or the program.

* `DEBUG message`  
  Tells git-annex to display the message if --debug is enabled.  
  (git-annex does not send a reply to this message.)
* `ERROR ErrorMsg`  
  Generic error. Can be sent at any time if things get too messed up to
  continue. When possible, use a more specific reply.  
  The program should exit after sending this, as git-annex will not talk to
  it any further. If the program receives an `ERROR` from git-annex, it can
  exit with its own `ERROR`.

## considerations for generating keys

See [[internals/key_format]] for how to format a key and details about the
parts of a key.

The backend name should match the name of the program, eg if the program
is git-annex-backend-XFOO, it should generate a key starting with "XFOO-".

The backend name (and program name) has to be all uppercase, and should be
reasonably short (max 10 bytes or so), and should be entirely ascii
alphanumerics. Eg, use similar names to other [[backends]]. It must not end
with "E" (see next paragraph for why).

git-annex will automatically also support an "E" variant of the backend,
which adds a filename extension to the end of the key. It does this
entirely transparently to the program, so while the repository may be using
XFOOE keys, the program will always generate and verify XFOO keys.

The key name is typically some kind of hash, but is not limited to a hash.
The length of it needs to be similar to the lengths of other git-annex
keys. Too long a key name will make it annoying to work with repositories
using them, or even cause problems due to filename length limits. 128 bytes
maximum, but shorter is better. It should be entirely ascii characters
in the set `A-Za-z0-9` and `-` is allowed, but other punctuation is not.

It's important that, if the program responds with
`ISCRYPTOGRAPHICALLYSECURE-YES`, the key name contains only a hash, and not
other data from some other source. That other data could be used to try to
mount a sha1 collision attack against git, by embedding colliding material
in the key name, where users are unlikely to notice it. While git has
several things that make sha1 collision attacks difficult, we don't want
this chink in the armor.

It's almost always a good idea to include the size field when generating a
key. The size does not need to be checked when verifying content, as
git-annex handles that for you. The only time it would make sense to omit
the size field is if the content of a key is not stable and might have
different sizes (like some URL keys do).

There's generally no reason to include the mtime field, and it should
never be verified when verifying content.

## program names must be unique

It's important that two different programs don't use the same name, because
that would result in bad behavior if the wrong program were used with a
repository with keys generated by the other program.

To avoid picking the same name, there is a list of known external backend
programs in [[backends]].

## signals

The program should not block SIGINT, or SIGTERM. Doing so may cause
git-annex to hang waiting on it to exit. Of course it's ok to catch those
signals and do some necessary cleanup before exiting.
