mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

reorg special remote docs

Browse source 
Made responses to git-annex requests be listed under each request.

This did lead to a little duplication since some replies are used for 2
requests, but it also makes it much clearer and easier to see how the
protocol works.

And, it makes each request self-contained, so they can be split out into
separate pages.
This commit is contained in:
Joey Hess 2019-05-28 14:00:05 -04:00
parent 63fa6b4a44
commit 4818e3606c
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
1 changed files with 126 additions and 147 deletions
Download patch file Download diff file Expand all files Collapse all files

273
doc/design/external_special_remote_protocol.mdwn
View file

@ -94,14 +94,12 @@ Now git-annex will send its next request.
Once git-annex is done with the special remote, it will close its stdin. Once git-annex is done with the special remote, it will close its stdin.
The special remote program can then exit. The special remote program can then exit.
## git-annex request messages ## git-annex request messages and replies
These are messages git-annex sends to the special remote program. These are messages git-annex sends to the special remote program.
None of these messages require an immediate reply. The special
remote can send any messages it likes while handling the requests.
Once the special remote has finished performing the request, it should Once the special remote has finished performing the request, it should
send one of the corresponding replies listed in the next section. send one of the listed replies.
The following requests *must* all be supported by the special remote. The following requests *must* all be supported by the special remote.
@ -112,132 +110,35 @@ The following requests *must* all be supported by the special remote.
different repositories, or as the configuration of a remote is changed. different repositories, or as the configuration of a remote is changed.
(Both `git annex initremote` and `git-annex enableremote` run this.) (Both `git annex initremote` and `git-annex enableremote` run this.)
So any one-time setup tasks should be done idempotently. So any one-time setup tasks should be done idempotently.
* `INITREMOTE-SUCCESS`
Indicates the INITREMOTE succeeded and the remote is ready to use.
* `INITREMOTE-FAILURE ErrorMsg`
Indicates that INITREMOTE failed.
* `PREPARE` * `PREPARE`
Tells the remote that it's time to prepare itself to be used. Tells the remote that it's time to prepare itself to be used.
Only a few requests for details about the remote can come before this. Only a few requests for details about the remote can come before this.
Those include EXTENSIONS, INITREMOTE, EXPORTSUPPORTED, and may later Those include EXTENSIONS, INITREMOTE, and EXPORTSUPPORTED, but others
include others. may be added later.
* `PREPARE-SUCCESS`
Sent as a response to PREPARE once the special remote is ready for use.
* `PREPARE-FAILURE ErrorMsg`
Sent as a response to PREPARE if the special remote cannot be used.
* `TRANSFER STORE|RETRIEVE Key File` * `TRANSFER STORE|RETRIEVE Key File`
Requests the transfer of a key. For STORE, the File is the file to upload; Requests the transfer of a key. For STORE, the File is the file to upload;
for RETRIEVE the File is where to store the download. for RETRIEVE the File is where to store the download.
Note that the File should not influence the filename used on the remote. Note that the File should not influence the filename used on the remote.
Note that in some cases, the File may contain whitespace. Note that in some cases, the File may contain whitespace.
Note that it's important that, while a Key is being stored, CHECKPRESENT It's important that, while a Key is being stored, `CHECKPRESENT`
not indicate it's present until all the data has been transferred. not indicate it's present until all the data has been transferred.
* `CHECKPRESENT Key` While the transfer is running, the remote can send any number of
Requests the remote to check if a key is present in it. `PROGRESS` messages. Once the transfer is done, it finishes by sending
* `REMOVE Key` one of these replies:
Requests the remote to remove a key's contents.
The following requests can optionally be supported. If not handled,
replying with `UNSUPPORTED-REQUEST` is acceptable.
* `EXTENSIONS List`
Sent to indicate protocol extensions which git-annex is capable
of using. The list is a space-delimited list of protocol extension
keywords. The remote can reply to this with its own EXTENSIONS list.
* `GETCOST`
Requests the remote to return a use cost. Higher costs are more expensive.
(See Config/Cost.hs for some standard costs.)
* `GETAVAILABILITY`
Requests the remote to send back an `AVAILABILITY` reply.
If the remote replies with `UNSUPPORTED-REQUEST`, its availability
is assumed to be global. So, only remotes that are only reachable
locally need to worry about implementing this.
* `CLAIMURL Url`
Asks the remote if it wishes to claim responsibility for downloading
an url. If so, the remote should send back an `CLAIMURL-SUCCESS` reply.
If not, it can send `CLAIMURL-FAILURE`.
* `CHECKURL Url`
Asks the remote to check if the url's content can currently be downloaded
(without downloading it). The remote replies with one of `CHECKURL-FAILURE`,
`CHECKURL-CONTENTS`, or `CHECKURL-MULTI`.
* `WHEREIS Key`
Asks the remote to provide additional information about ways to access
the content of a key stored in it, such as eg, public urls.
This will be displayed to the user by eg, `git annex whereis`. The remote
replies with `WHEREIS-SUCCESS` or `WHEREIS-FAILURE`.
Note that users expect `git annex whereis` to run fast, without eg,
network access.
This is not needed when `SETURIPRESENT` is used, since such uris are
automatically displayed by `git annex whereis`.
* `GETINFO`
Requests the remote to send some information describing its
configuration, for display by `git annex info`.
Reply with a series of `INFOFIELD` each followed by `INFOVALUE`,
and concluded with `INFOEND`.
* `EXPORTSUPPORTED`
Used to check if a special remote supports exports. The remote
responds with either `EXPORTSUPPORTED-SUCCESS` or
`EXPORTSUPPORTED-FAILURE`. Note that this request may be made before
or after `PREPARE`.
* `EXPORT Name`
Comes immediately before each of the following export-related requests,
specifying the name of the exported file. It will be in the form
of a relative path, and may contain path separators, whitespace,
and other special characters.
No response is made to this message.
* `TRANSFEREXPORT STORE|RETRIEVE Key File`
Requests the transfer of a File on local disk to or from the previously
provided Name on the special remote.
Note that it's important that, while a file is being stored,
CHECKPRESENTEXPORT not indicate it's present until all the data has
been transferred.
The remote responds with either `TRANSFER-SUCCESS` or
`TRANSFER-FAILURE`, and a remote where exports do not make sense
may always fail.
* `CHECKPRESENTEXPORT Key`
Requests the remote to check if the previously provided Name is present
in it.
The remote responds with `CHECKPRESENT-SUCCESS`, `CHECKPRESENT-FAILURE`,
or `CHECKPRESENT-UNKNOWN`.
* `REMOVEEXPORT Key`
Requests the remote to remove content stored by `TRANSFEREXPORT`
with the previously provided Name.
The remote responds with either `REMOVE-SUCCESS` or
`REMOVE-FAILURE`.
If the content was already not present in the remote, it should
respond with `REMOVE-SUCCESS`.
* `REMOVEEXPORTDIRECTORY Directory`
Requests the remote remove an exported directory.
If the remote does not use directories, or REMOVEEXPORT cleans up
directories that are empty, this does not need to be implemented.
The directory will be in the form of a relative path, and may contain path
separators, whitespace, and other special characters.
Typically the directory will be empty, but it could possibly contain
files or other directories, and it's ok to remove those.
The remote responds with either `REMOVEEXPORTDIRECTORY-SUCCESS`
or `REMOVEEXPORTDIRECTORY-FAILURE`.
Should not fail if the directory was already removed.
* `RENAMEEXPORT Key NewName`
Requests the remote rename a file stored on it from the previously
provided Name to the NewName.
The remote responds with `RENAMEEXPORT-SUCCESS` or
`RENAMEEXPORT-FAILURE` or with `UNSUPPORTED-REQUEST` if renaming is not
supported.
To support old external special remote programs that have not been updated
to support exports, git-annex will need to handle an `ERROR` response
when using any of the above.
More optional requests may be added, without changing the protocol version,
so if an unknown request is seen, reply with `UNSUPPORTED-REQUEST`.
## special remote replies
These should be sent only in response to the git-annex request messages.
They do not have to be sent immediately after the request; the special
remote can send its own messages (listed in the next section below)
while it's handling a request.
* `PREPARE-SUCCESS`
Sent as a response to PREPARE once the special remote is ready for use.
* `PREPARE-FAILURE ErrorMsg`
Sent as a response to PREPARE if the special remote cannot be used.
* `TRANSFER-SUCCESS STORE|RETRIEVE Key` * `TRANSFER-SUCCESS STORE|RETRIEVE Key`
Indicates the transfer completed successfully. Indicates the transfer completed successfully.
* `TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg` * `TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg`
Indicates the transfer failed. Indicates the transfer failed.
* `CHECKPRESENT Key`
Requests the remote to check if a key is present in it.
* `CHECKPRESENT-SUCCESS Key` * `CHECKPRESENT-SUCCESS Key`
Indicates that a key has been positively verified to be present in the Indicates that a key has been positively verified to be present in the
remote. remote.
@ -247,28 +148,48 @@ while it's handling a request.
* `CHECKPRESENT-UNKNOWN Key ErrorMsg` * `CHECKPRESENT-UNKNOWN Key ErrorMsg`
Indicates that it is not currently possible to verify if the key is Indicates that it is not currently possible to verify if the key is
present in the remote. (Perhaps the remote cannot be contacted.) present in the remote. (Perhaps the remote cannot be contacted.)
* `REMOVE Key`
Requests the remote to remove a key's contents.
* `REMOVE-SUCCESS Key` * `REMOVE-SUCCESS Key`
Indicates the key has been removed from the remote. May be returned if Indicates the key has been removed from the remote. May be returned if
the remote didn't have the key at the point removal was requested. the remote didn't have the key at the point removal was requested.
* `REMOVE-FAILURE Key ErrorMsg` * `REMOVE-FAILURE Key ErrorMsg`
Indicates that the key was unable to be removed from the remote. Indicates that the key was unable to be removed from the remote.
The following requests can optionally be supported. If not supported,
the special remote can reply with `UNSUPPORTED-REQUEST`.
* `EXTENSIONS List`
Sent to indicate protocol extensions which git-annex is capable
of using. The list is a space-delimited list of protocol extension
keywords. The remote can reply to this with its own EXTENSIONS list.
* `EXTENSIONS List` * `EXTENSIONS List`
Sent in response to a EXTENSIONS request, the List could be used to indicate Sent in response to a EXTENSIONS request, the List could be used to indicate
protocol extensions that the special remote uses, but there are currently protocol extensions that the special remote uses, but there are currently
no such extensions, so the List is empty. no such extensions.
* `GETCOST`
Requests the remote to return a use cost. Higher costs are more expensive.
(See Config/Cost.hs for some standard costs.)
* `COST Int` * `COST Int`
Indicates the cost of the remote. Indicates the cost of the remote.
* `GETAVAILABILITY`
Asks the remote if it is locally or globally available.
(Ie stored in the cloud vs on a local disk.)
If the remote replies with `UNSUPPORTED-REQUEST`, its availability
is assumed to be global. So, only remotes that are only reachable
locally need to worry about implementing this.
* `AVAILABILITY GLOBAL|LOCAL` * `AVAILABILITY GLOBAL|LOCAL`
Indicates if the remote is globally or only locally available. Indicates if the remote is globally or only locally available.
(Ie stored in the cloud vs on a local disk.) * `CLAIMURL Url`
* `INITREMOTE-SUCCESS` Asks the remote if it wishes to claim responsibility for downloading
Indicates the INITREMOTE succeeded and the remote is ready to use. an url.
* `INITREMOTE-FAILURE ErrorMsg`
Indicates that INITREMOTE failed.
* `CLAIMURL-SUCCESS` * `CLAIMURL-SUCCESS`
Indicates that the CLAIMURL url will be handled by this remote. Indicates that the CLAIMURL url will be handled by this remote.
* `CLAIMURL-FAILURE` * `CLAIMURL-FAILURE`
Indicates that the CLAIMURL url wil not be handled by this remote. Indicates that the CLAIMURL url wil not be handled by this remote.
* `CHECKURL Url`
Asks the remote to check if the url's content can currently be downloaded
(without downloading it).
* `CHECKURL-CONTENTS Size|UNKNOWN Filename` * `CHECKURL-CONTENTS Size|UNKNOWN Filename`
Indicates that the requested url has been verified to exist. Indicates that the requested url has been verified to exist.
The Size is the size in bytes, or use "UNKNOWN" if the size could not be The Size is the size in bytes, or use "UNKNOWN" if the size could not be
@ -284,45 +205,103 @@ while it's handling a request.
can contain spaces. can contain spaces.
* `CHECKURL-FAILURE` * `CHECKURL-FAILURE`
Indicates that the requested url could not be accessed. Indicates that the requested url could not be accessed.
* `WHEREIS Key`
Asks the remote to provide additional information about ways to access
the content of a key stored in it, such as eg, public urls.
This will be displayed to the user by eg, `git annex whereis`.
Note that users expect `git annex whereis` to run fast, without eg,
network access.
* `WHEREIS-SUCCESS String` * `WHEREIS-SUCCESS String`
Indicates a location of a key. Typically an url, the string can Indicates a location of a key. Typically an url, the string can
be anything that it makes sense to display to the user about content be anything that it makes sense to display to the user about content
stored in the special remote. stored in the special remote.
* `WHEREIS-FAILURE` * `WHEREIS-FAILURE`
Indicates that no location is known for a key. Indicates that no location is known for a key.
* `INFOFIELD` / `INFOVALUE` / `INFOEND` This is not needed when `SETURIPRESENT` is used, since such uris are
Reply to a GETINFO request. This can be used to add info about anything, automatically displayed by `git annex whereis`.
but things like an url to the remote, or details of the remote's * `GETINFO`
configuration are typical. It should not include any sensitive Requests the remote to send some information describing its
information like passwords, since it will be displayed to the user's configuration, for display by `git annex info`. A block of responses
screen. can be made to this, which must always end with `INFOEND`.
* `INFOFIELD Name`
There can be zero or more `INFOFIELD` messages, each containing the name of Gives the name of an info field. The name can be anything you want to
a field, and each is immediately followed by an `INFOVALUE` message be displayed to the user. Must be immediately followed by `INFOVALUE`.
containing its value. The sequence is concluded by `INFOEND`. For example: * `INFOVALUE Value`
Gives the value of an info field.
INFOFIELD repository location * `INFOEND`
INFOVALUE http://example.com/repo/ Indicates the end of the response block.
INFOFIELD datacenter * `EXPORTSUPPORTED`
INFOVALUE Antarctica Used to check if a special remote supports exports.
INFOEND Note that this request may be made before or after `PREPARE`.
* `EXPORTSUPPORTED-SUCCESS` * `EXPORTSUPPORTED-SUCCESS`
Indicates that it makes sense to use this special remote as an export. Indicates that it makes sense to use this special remote as an export.
* `EXPORTSUPPORTED-FAILURE` * `EXPORTSUPPORTED-FAILURE`
Indicates that it does not make sense to use this special remote as an Indicates that it does not make sense to use this special remote as an
export. export.
* `RENAMEEXPORT-SUCCESS Key` * `EXPORT Name`
Indicates that a `RENAMEEXPORT` was done successfully. Comes immediately before each of the following export-related requests,
* `RENAMEEXPORT-FAILURE Key` specifying the name of the exported file. It will be in the form
Indicates that a `RENAMEEXPORT` failed for whatever reason. of a relative path, and may contain path separators, whitespace,
and other special characters.
No response is made to this message.
* `TRANSFEREXPORT STORE|RETRIEVE Key File`
Requests the transfer of a File on local disk to or from the previously
provided Name on the special remote.
Note that it's important that, while a file is being stored,
CHECKPRESENTEXPORT not indicate it's present until all the data has
been transferred.
While the transfer is running, the remote can send any number of
`PROGRESS` messages. Once the transfer is complete, it finishes by
sending one of these replies:
* `TRANSFER-SUCCESS STORE|RETRIEVE Key`
Indicates the transfer completed successfully.
* `TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg`
Indicates the transfer failed.
* `CHECKPRESENTEXPORT Key`
Requests the remote to check if the previously provided Name is present
in it.
* `CHECKPRESENT-SUCCESS Key`
Indicates that a content has been positively verified to be present in the
remote.
* `CHECKPRESENT-FAILURE Key`
Indicates that a contents has been positively verified to not be present in the
remote.
* `CHECKPRESENT-UNKNOWN Key ErrorMsg`
Indicates that it is not currently possible to verify if content is
present in the remote. (Perhaps the remote cannot be contacted.)
* `REMOVEEXPORT Key`
Requests the remote to remove content stored by `TRANSFEREXPORT`
with the previously provided Name.
* `REMOVE-SUCCESS Key`
Indicates the content has been removed from the remote. May be returned when
the content was already not present.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the content was unable to be removed from the remote.
* `REMOVEEXPORTDIRECTORY Directory`
Requests the remote remove an exported directory.
If the remote does not use directories, or REMOVEEXPORT cleans up
directories that are empty, this does not need to be implemented.
The directory will be in the form of a relative path, and may contain path
separators, whitespace, and other special characters.
Typically the directory will be empty, but it could possibly contain
files or other directories, and it's ok to remove those.
* `REMOVEEXPORTDIRECTORY-SUCCESS` * `REMOVEEXPORTDIRECTORY-SUCCESS`
Indicates that a `REMOVEEXPORTDIRECTORY` was done successfully. Indicates that a `REMOVEEXPORTDIRECTORY` was done successfully.
* `REMOVEEXPORTDIRECTORY-FAILURE` * `REMOVEEXPORTDIRECTORY-FAILURE`
Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason. Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason.
* `UNSUPPORTED-REQUEST` Should not be returned if the directory did not exist.
Indicates that the special remote does not know how to handle a request, * `RENAMEEXPORT Key NewName`
or cannot handle the request. Requests the remote rename a file stored on it from the previously
provided Name to the NewName. Remotes that support exports but not
renaming do not need to implement this.
* `RENAMEEXPORT-SUCCESS Key`
Indicates that a `RENAMEEXPORT` was done successfully.
* `RENAMEEXPORT-FAILURE Key`
Indicates that a `RENAMEEXPORT` failed for whatever reason.
More optional requests may be added, without changing the protocol version,
so if an unknown request is seen, don't crash, just reply with
`UNSUPPORTED-REQUEST`.
## special remote messages ## special remote messages