mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc/design/external_special_remote_protocol/async_appendix.mdwn

100 lines
2.8 KiB
Text
Raw Normal View History

draft async extension
2020-08-11 20:42:09 +00:00
(This is a draft and not implemented yet.)
This is an appendix to the [[external_special_remote_protocol]].
[[!toc]]
## introduction
Normally, an external special remote can only be used to do one thing at a
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
time, and when git-annex has concurrency enabled, it will start up multiple
draft async extension
2020-08-11 20:42:09 +00:00
processes for the same external special remote.
This extension lets a single external special remote process handle
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
multiple concurrent jobs, which can be useful if multiple processes
draft async extension
2020-08-11 20:42:09 +00:00
would use too many resources, or if it can be better coordinated using a
single process.
## protocol overview
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
As usual, the protocol starts by the external special remote sending
the version of the protocol it's using.
VERSION 1
draft async extension
2020-08-11 20:42:09 +00:00
This extension is negotiated by git-annex sending an `EXTENSIONS` message
that includes `ASYNC`, and the external special remote responding in kind.
EXTENSIONS INFO ASYNC
EXTENSIONS ASYNC
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
From this point forward, every message in the protocol is tagged with a job
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
number, by prefixing it with "J n".
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
As usual, the first message git-annex sends is generally PREPARE:
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
J 1 PREPARE
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
Rather than just responding PREPARE-SUCCESS, the job number has to be
included in the reply:
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
J 1 PREPARE-SUCCESS
draft async extension
2020-08-11 20:42:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
Suppose git-annex wants to make some transfers. It can request several
at the same time, using different job numbers:
draft async extension
2020-08-11 20:42:09 +00:00
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
J 1 TRANSFER RETRIEVE Key1 file1
J 2 TRANSFER RETRIEVE Key2 file2
draft async extension
2020-08-11 20:42:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
The special remote can now perform both transfers at the same time.
If it sends PROGRESS messages for these transfers, they have to be tagged
with the job number:
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
J 1 PROGRESS 10
J 2 PROGRESS 500
J 1 PROGRESS 20
draft async extension
2020-08-11 20:42:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
The special remote can also send messages that query git-annex for some
information. These messages and the reply will also be tagged with a job
number.
draft async extension
2020-08-11 20:42:09 +00:00
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
J 1 GETCONFIG url
J 3 RETRIEVE Key3 file3
J 1 VALUE http://example.com/
draft async extension
2020-08-11 20:42:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
One transfers are done, the special remote sends `TRANSFER-SUCCESS` tagged
with the job number.
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
J 2 TRANSFER-SUCCESS RETRIEVE Key2
J 1 PROGRESS 100
J 1 TRANSFER-SUCCESS RETRIEVE Key1
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
Lots of different jobs can be requested at the same time.
J 4 CHECKPRESENT Key3
J 5 CHECKPRESENT Key4
J 6 REMOVE Key3
J 4 CHECKPRESENT-SUCCESS Key3
J 6 REMOVE-SUCCESS Key3
J 5 CHECKPRESENT-FAILURE Key4
An example of sending multiple replies to a request is `LISTCONFIGS`, eg:
J 7 LISTCONFIGS
J 7 CONFIG foo some config
J 7 CONFIG bar other config
J 7 CONFIGEND
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
## notes
There will generally be one job for each thread that git-annex runs
concurrently, so around the same number as the -J value, although in some
cases git-annex does more concurrent operations than the -J value.
`PREPARE` is sent only once per run of a special remote
program, and despite being tagged with a job number, it should prepare the
special remote to run that and any other jobs.
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
`ERROR` should not be tagged with a job number if either git-annex
or the special remote needs to send it.
Reference in a new issue Copy permalink