198b709561
TVars were not updated atomically, which was ok when each thread got its own External that was the only thing using these TVars. But, with the async extension, several External instances can share the same var, so it needs to be a TMVar to avoid read/write conflicts. In particular, this makes PREPARE only be sent once.
99 lines
2.8 KiB
Markdown
99 lines
2.8 KiB
Markdown
(This is a draft and not implemented yet.)
|
|
|
|
This is an appendix to the [[external_special_remote_protocol]].
|
|
|
|
[[!toc]]
|
|
|
|
## introduction
|
|
|
|
Normally, an external special remote can only be used to do one thing at a
|
|
time, and when git-annex has concurrency enabled, it will start up multiple
|
|
processes for the same external special remote.
|
|
|
|
This extension lets a single external special remote process handle
|
|
multiple concurrent jobs, which can be useful if multiple processes
|
|
would use too many resources, or if it can be better coordinated using a
|
|
single process.
|
|
|
|
## protocol overview
|
|
|
|
As usual, the protocol starts by the external special remote sending
|
|
the version of the protocol it's using.
|
|
|
|
VERSION 1
|
|
|
|
This extension is negotiated by git-annex sending an `EXTENSIONS` message
|
|
that includes `ASYNC`, and the external special remote responding in kind.
|
|
|
|
EXTENSIONS INFO ASYNC
|
|
EXTENSIONS ASYNC
|
|
|
|
From this point forward, every message in the protocol is tagged with a job
|
|
number, by prefixing it with "J n".
|
|
|
|
As usual, the first message git-annex sends is generally PREPARE:
|
|
|
|
J 1 PREPARE
|
|
|
|
Rather than just responding PREPARE-SUCCESS, the job number has to be
|
|
included in the reply:
|
|
|
|
J 1 PREPARE-SUCCESS
|
|
|
|
Suppose git-annex wants to make some transfers. It can request several
|
|
at the same time, using different job numbers:
|
|
|
|
J 1 TRANSFER RETRIEVE Key1 file1
|
|
J 2 TRANSFER RETRIEVE Key2 file2
|
|
|
|
The special remote can now perform both transfers at the same time.
|
|
If it sends PROGRESS messages for these transfers, they have to be tagged
|
|
with the job number:
|
|
|
|
J 1 PROGRESS 10
|
|
J 2 PROGRESS 500
|
|
J 1 PROGRESS 20
|
|
|
|
The special remote can also send messages that query git-annex for some
|
|
information. These messages and the reply will also be tagged with a job
|
|
number.
|
|
|
|
J 1 GETCONFIG url
|
|
J 3 RETRIEVE Key3 file3
|
|
J 1 VALUE http://example.com/
|
|
|
|
One transfers are done, the special remote sends `TRANSFER-SUCCESS` tagged
|
|
with the job number.
|
|
|
|
J 2 TRANSFER-SUCCESS RETRIEVE Key2
|
|
J 1 PROGRESS 100
|
|
J 1 TRANSFER-SUCCESS RETRIEVE Key1
|
|
|
|
Lots of different jobs can be requested at the same time.
|
|
|
|
J 4 CHECKPRESENT Key3
|
|
J 5 CHECKPRESENT Key4
|
|
J 6 REMOVE Key3
|
|
J 4 CHECKPRESENT-SUCCESS Key3
|
|
J 6 REMOVE-SUCCESS Key3
|
|
J 5 CHECKPRESENT-FAILURE Key4
|
|
|
|
An example of sending multiple replies to a request is `LISTCONFIGS`, eg:
|
|
|
|
J 7 LISTCONFIGS
|
|
J 7 CONFIG foo some config
|
|
J 7 CONFIG bar other config
|
|
J 7 CONFIGEND
|
|
|
|
## notes
|
|
|
|
There will be one job number for each thread that git-annex runs
|
|
concurrently, so around the same number as the -J value, although in some
|
|
cases git-annex does more concurrent operations than the -J value.
|
|
|
|
`PREPARE` is sent only once per run of a special remote
|
|
program, and despite being tagged with a job number, it should prepare the
|
|
special remote to run that and any other jobs.
|
|
|
|
`ERROR` should not be tagged with a job number if either git-annex
|
|
or the special remote needs to send it.
|