15706e6991
Receive loop looks right. Still need the send loop. And, a complication is that some messages git-annex sends need to be wrapped in REPLY_ASYNC, while others do not. So will probably need to split externalSend into two.
134 lines
4.7 KiB
Markdown
134 lines
4.7 KiB
Markdown
(This is a draft and not implemented yet.)
|
|
|
|
This is an appendix to the [[external_special_remote_protocol]].
|
|
|
|
[[!toc]]
|
|
|
|
## introduction
|
|
|
|
Normally, an external special remote can only be used to do one thing at a
|
|
time. When git-annex has concurrency enabled, it will start up multiple
|
|
processes for the same external special remote.
|
|
|
|
This extension lets a single external special remote process handle
|
|
multiple concurrent requests, which can be useful if multiple processes
|
|
would use too many resources, or if it can be better coordinated using a
|
|
single process.
|
|
|
|
## protocol overview
|
|
|
|
As usual, the protocol starts by the external special remote sending
|
|
the version of the protocol it's using.
|
|
|
|
VERSION 1
|
|
|
|
This extension is negotiated by git-annex sending an `EXTENSIONS` message
|
|
that includes `ASYNC`, and the external special remote responding in kind.
|
|
|
|
EXTENSIONS INFO ASYNC
|
|
EXTENSIONS ASYNC
|
|
|
|
From this point forward, *everything* that the external special remote
|
|
has to be wrapped in the async protocol. Messages git-annex sends are
|
|
unchanged.
|
|
|
|
Generally the first message git-annex sends will be PREPARE.
|
|
|
|
PREPARE
|
|
|
|
Rather than just responding PREPARE-SUCCESS, it has to be wrapped
|
|
in the async protocol:
|
|
|
|
RESULT-ASYNC PREPARE-SUCCESS
|
|
|
|
Suppose git-annex wants to make some transfers. So it sends:
|
|
|
|
TRANSFER RETRIEVE Key1 file1
|
|
|
|
The special remote should respond with an unique identifier for this
|
|
async job that it's going to start. The identifier can
|
|
be anything you want to use, but an incrementing number is a
|
|
reasonable choice. (The Key itself is not a good choice, because git-annex
|
|
could make different requests involving the same Key.)
|
|
|
|
START-ASYNC 1
|
|
|
|
Once that's sent, git-annex can send its next request immediately,
|
|
while that transfer is still running. For example, it might request a
|
|
second transfer, and the special remote can reply when it's started that
|
|
transfer too:
|
|
|
|
TRANSFER RETRIEVE 2 file2
|
|
START-ASYNC 2
|
|
|
|
If it needs to query git-annex for some information, the special remote
|
|
can use `ASYNC` to send a message, and wait for git-annex to reply
|
|
in a `REPLY-ASYNC` message:
|
|
|
|
ASYNC 1 GETCONFIG url
|
|
REPLY-ASYNC 1 VALUE http://example.com/
|
|
|
|
To indicate progress of transfers, the special remote can send
|
|
`ASYNC` messages, wrapping the usual PROGRESS messages:
|
|
|
|
ASYNC 1 PROGRESS 10
|
|
ASYNC 2 PROGRESS 500
|
|
ASYNC 1 PROGRESS 20
|
|
|
|
Once a transfer is done, the special remote indicates this with an
|
|
`END-ASYNC` message, wrapping the usual `TRANSFER-SUCCESS` or
|
|
`TRANSFER-FAILURE` message:
|
|
|
|
END-ASYNC 2 TRANSFER-SUCCESS RETRIEVE Key2
|
|
ASYNC Key1 PROGRESS 100
|
|
END-ASYNC 1 TRANSFER-SUCCESS RETRIEVE Key1
|
|
|
|
Not only transfers, but everything the special remote sends to git-annex
|
|
has to be wrapped in the async protocol.
|
|
|
|
CHECKPRESENT Key3
|
|
START-ASYNC 3
|
|
CHECKPRESENT Key4
|
|
START-ASYNC 4
|
|
END-ASYNC 3 CHECKPRESENT-SUCCESS Key3
|
|
REMOVE Key3
|
|
END-ASYNC 4 CHECKPRESENT-FAILURE Key4
|
|
START_ASYNC 5
|
|
END-ASYNC 5 REMOVE-SUCCESS Key3
|
|
|
|
## added messages
|
|
|
|
Here's the details about the additions to the protocol.
|
|
|
|
* `START-ASYNC JobId`
|
|
This (or `RESULT-ASYNC` must be sent in response to all requests
|
|
git-annex sends after `EXTENSIONS` has been used to negotiate the
|
|
async protocol.
|
|
The JobId is a unique value, typically an incrementing number.
|
|
This does not need to be sent immediately after git-annex sends a request;
|
|
other messages can be sent in between. But the next START-ASYNC git-annex sees
|
|
after sending a request tells it the JobId that will be used for that request.
|
|
* `END-ASYNC JobId ReplyMsg`
|
|
Indicates that an async job is complete. The ReplyMsg indicates the result
|
|
of the job, and is anything that would be sent as a protocol reply in the
|
|
non-async protocol.
|
|
After this, the JobId is not in use, an indeed the same value could be
|
|
reused by a new `START-ASYNC` if desired.
|
|
* `RESULT-ASYNC ReplyMsg`
|
|
This is the same as sending `START-ASYNC` immediately followed by
|
|
`END-ASYNC`. This is often used to respond to `PREPARE`, `LISTCONFIGS`,
|
|
and other things that are trivial or just don't need to be handled async.
|
|
* `ASYNC JobId InfoMsg`
|
|
Used to send any of the [special remote messages](https://git-annex.branchable.com/design/external_special_remote_protocol/#index5h2)
|
|
to git-annex.
|
|
Often used to send `PROGRESS`, but can also be used for other messages,
|
|
including ones that git-annex sends a reply to. When git-annex does send
|
|
a reply,
|
|
it will be wrapped in `REPLY-ASYNC`.
|
|
Can be sent at any time aftwr `START-ASYNC` and before `END-ASYNC` for
|
|
the JobId in question.
|
|
* `REPLY-ASYNC JobId Reply`
|
|
Sent by git-annex when `ASYNC` has been sent and the message generated
|
|
a reply. Note that this may not be the next message received from
|
|
git-annex immediately after sending an `ASYNC` request.
|
|
|