git-annex/doc/design/external_special_remote_protocol.mdwn

See [[todo/support_for_writing_external_special_remotes]] for motivation.

This is a design for a protocol to be used to communicate between git-annex
and a program implementing an external special remote.

The program has a name like `git-annex-remote-$bar`. When
`git annex initremote foo type=$bar` is run, git-annex finds the
appropriate program in PATH.

The program is started by git-annex when it needs to access the special
remote, and may be left running for a long period of time. This allows
it to perform expensive setup tasks, etc. Note that git-annex may choose to
start multiple instances of the program (eg, when multiple git-annex
commands are run concurrently in a repository).

Communication is via the programs stdin and stdout. Therefore, the program
must avoid doing any prompting, or outputting anything like eg, progress to
stdout. (Such stuff can be sent to stderr instead.)

The protocol is line based. Messages are sent in either direction, from
git-annex to the program, and from the program to git-annex. No immediate
reply is made to any message, instead a later message can be sent to reply.

## example

For example, git-annex might request that a key be sent to the
remote (Key will be replaced with the key, and File with a file that has
the content to send):

	TRANSFER STORE Key File

Any number of messages can be sent back and forth while that upload
is going on. A common message the program would send is to tell the
progress of the upload (in bytes):

	PROGRESS STORE Key 10240
	PROGRESS STORE Key 20480

Once the file has been sent, the program can reply with the result:

	TRANSFER-SUCCESS STORE Key

## git-annex messages

These are the messages git-annex may send to the special remote program.

* `CONFIGURE KEY=VALUE ...`  
  Tells the remote its configuration. Any arbitrary KEY(s) can be passed.
  Only run once, at startup.
* `INITREMOTE`  
  Request that the remote be initialized. CONFIGURE will be passed first.
  Note that this may be run repeatedly, as a remote is initialized in
  different repositories, or as the configuration of a remote is changed.
* `GETCOST`  
  Requests the remote return a use cost. Higher costs are more expensive.
  (See Config/Cost.hs for some standard costs.)
* `TRANSFER STORE|RETRIEVE Key File`  
  Requests the transfer of a key. For Send, the File is the file to upload;
  for Receive the File is where to store the download. Note that the File
  should not influence the filename used on the remote. The filename used
  should be derived from the Key.  
  Multiple transfers might be requested by git-annex, but it's fine for the 
  program to serialize them and only do one at a time.
* `HAS Key`  
  Requests the remote check if a key is present in it.
* `REMOVE Key`  
  Requests the remote remove a key's contents.
  

## special remote messages

These are the messages the special remote program can send back to
git-annex.

* `VERSION Int`  
  Supported protocol version. Current version is 0. Must be sent first
  thing at startup, as until it sees this git-annex does not know how to
  talk with the special remote program!
* `ERROR ErrorMsg`  
  Generic error. Can be sent at any time if things get messed up.
  It would be a good idea to send this if git-annex sends a command
  you do not support. The program should exit after sending this, as
  git-annex will not talk to it any further.
* `TRANSFER-SUCCESS STORE|RETRIEVE Key`  
  Indicates the transfer completed successfully.
* `TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg`  
  Indicates the transfer failed.
* `PROGRESS STORE|RETRIEVE Key Int`  
  Indicates the current progress of the transfer. May be repeated any
  number of times during the transfer process. This is highly recommended
  for STORE. (It is not necessary for RETRIEVE.)
* `HAS-SUCCESS Key`  
  Indicates that a key has been positively verified to be present in the
  remote.
* `HAS-FAILURE Key`  
  Indicates that a key has been positively verified to not be present in the
  remote.
* `HAS-UNKNOWN Key ErrorMsg`  
  Indicates that it is not currently possible to verify if the key is
  present in the remote. (Perhaps the remote cannot be contacted.)
* `REMOVE-SUCCESS Key`  
  Indicates the key has been removed from the remote. May be returned if
  the remote didn't have the key at the point removal was requested.
* `REMOVE-FAILURE Key`  
  Indicates that the key was unable to be removed from the remote.
* `COST Int`  
  Indicates the cost of the remote.
* `COST-UNKNOWN`  
  Indicates the remote has no opinion of its cost.
* `CONFIGURE-SUCCESS`  
  Indicates the CONFIGURE provided an acceptable configuration.
* `CONFIGURE-FAILURE ErrorMsg`  
  Indicates that CONFIGURE provided a bad configuration.
* `INITREMOTE-SUCCESS KEY=VALUE ...`  
  Indicates the INITREMOTE succeeded and the remote is ready to use.
  The keys and values can optionally be returned. They will be added
  to the existing configuration of the remote (and may change existing
  values in it), and sent back the next time it calls CONFIGURE.
* `INITREMOTE-FAILURE ErrorMsg`  
  Indicates that INITREMOTE failed.

## Simple shell example

[[!format sh """
#!/bin/sh
set -e

send () {
	echo "$@"
}

send VERSION 0

while read line; do
	set -- $line
	case "$1" in
		CONFIGURE)
			send CONFIGURE-SCCESS
		;;
		INITREMOTE)
			send INITREMOTE-SUCCESS
		;;
		GETCOST)
			send COST-UNKNOWN
		;;
		TRANSFER)
			key="$3"
			file="$4"
			case "$2" in
				STORE)
					# XXX upload file here
					# XXX when possible, send PROGRESS
					send TRANSFER-SUCCESS STORE "$key"
				;;
				RETRIEVE)
					# XXX download file here
					send TRANSFER-SUCCESS RETRIEVE "$key"
				;;
				
			esac
		;;
		HAS)
			key="$2"
			send HAS-UNKNOWN "$key" "not implemented"
		;;
		REMOVE)
			key="$2"
			# XXX remove key here
			send REMOVE-SUCCESS "$key"
		;;
		*)
			send ERROR "unknown command received: $line"
			exit 1
		;;
	esac	
done
"""]]

## TODO

* Communicate when the network connection may have changed, so long-running
  remotes can reconnect.
* Provide a way for remotes to set/get the content of a per-key
  file in the git-annex branch. Needed for eg, storing urls, or access keys
  used to retrieve a given key.
* Support for splitting files into chunks.
* git-annex hash directory lookup for a key?
* Use same verbs as used in special remote interface (instead of different
  verbs used in Types.Remote).