98 lines
3 KiB
Markdown
98 lines
3 KiB
Markdown
This is an appendix to the [[external_special_remote_protocol]].
|
|
|
|
[[!toc]]
|
|
|
|
## introduction
|
|
|
|
Normally, an external special remote can only be used to do one thing at a
|
|
time, and when git-annex has concurrency enabled, it will start up multiple
|
|
processes for the same external special remote.
|
|
|
|
The `ASYNC` extension lets a single external special remote process handle
|
|
multiple concurrent jobs, which can be useful if multiple processes
|
|
would use too many resources, or if it can be better coordinated using a
|
|
single process.
|
|
|
|
## protocol overview
|
|
|
|
As usual, the protocol starts by the external special remote sending
|
|
the version of the protocol it's using.
|
|
|
|
VERSION 1
|
|
|
|
This extension is negotiated by git-annex sending an `EXTENSIONS` message
|
|
that includes `ASYNC`, and the external special remote responding in kind.
|
|
|
|
EXTENSIONS INFO ASYNC
|
|
EXTENSIONS ASYNC
|
|
|
|
(Older versions of git-annex will not include `ASYNC` in their extensions
|
|
list. To support them, it's a good idea for the external special remote to
|
|
fall back to using the regular protocol.)
|
|
|
|
Once the extension is negotiated, messages in the protocol are
|
|
tagged with a job number, by prefixing them with "J n".
|
|
|
|
As usual, the first message git-annex sends is generally PREPARE,
|
|
which gets tagged with a job number:
|
|
|
|
J 1 PREPARE
|
|
|
|
Rather than just responding PREPARE-SUCCESS, the job number has to be
|
|
included in the reply:
|
|
|
|
J 1 PREPARE-SUCCESS
|
|
|
|
Suppose git-annex wants to make some transfers. It can request several
|
|
at the same time, using different job numbers:
|
|
|
|
J 1 TRANSFER RETRIEVE Key1 file1
|
|
J 2 TRANSFER RETRIEVE Key2 file2
|
|
|
|
The special remote can now perform both transfers at the same time.
|
|
If it sends PROGRESS messages for these transfers, they have to be tagged
|
|
with the job numbers:
|
|
|
|
J 1 PROGRESS 10
|
|
J 2 PROGRESS 500
|
|
J 1 PROGRESS 20
|
|
|
|
The special remote can also send messages that query git-annex for some
|
|
information. These messages and the reply will also be tagged with a job
|
|
number.
|
|
|
|
J 1 GETCONFIG url
|
|
J 3 RETRIEVE Key3 file3
|
|
J 1 VALUE http://example.com/
|
|
|
|
One transfers are done, the special remote sends `TRANSFER-SUCCESS` tagged
|
|
with the job number.
|
|
|
|
J 2 TRANSFER-SUCCESS RETRIEVE Key2
|
|
J 1 PROGRESS 100
|
|
J 1 TRANSFER-SUCCESS RETRIEVE Key1
|
|
|
|
Lots of different jobs can be requested at the same time.
|
|
|
|
J 4 CHECKPRESENT Key3
|
|
J 5 CHECKPRESENT Key4
|
|
J 6 REMOVE Key5
|
|
J 4 CHECKPRESENT-SUCCESS Key3
|
|
J 6 REMOVE-SUCCESS Key5
|
|
J 5 CHECKPRESENT-FAILURE Key4
|
|
|
|
## notes
|
|
|
|
There will be one job number for each thread that git-annex runs
|
|
concurrently, so around the same number as the -J value, although in some
|
|
cases git-annex does more concurrent operations than the -J value.
|
|
|
|
`PREPARE` is sent only once per run of a special remote
|
|
program, and despite being tagged with a job number, it should prepare the
|
|
special remote to run that and any other jobs.
|
|
|
|
`ERROR` should not be tagged with a job number if either git-annex
|
|
or the special remote needs to send it.
|
|
|
|
`VERSION`, `EXTENSIONS` and `ERROR` are the only protocol messages
|
|
that do not get tagged with a job number.
|