mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc/design/external_special_remote_protocol/async_appendix.mdwn

99 lines
3 KiB
Text
Raw Normal View History

draft async extension
2020-08-11 20:42:09 +00:00
This is an appendix to the [[external_special_remote_protocol]].
[[!toc]]
## introduction
Normally, an external special remote can only be used to do one thing at a
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
time, and when git-annex has concurrency enabled, it will start up multiple
draft async extension
2020-08-11 20:42:09 +00:00
processes for the same external special remote.
async extension done
2020-08-14 19:24:34 +00:00
The `ASYNC` extension lets a single external special remote process handle
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
multiple concurrent jobs, which can be useful if multiple processes
draft async extension
2020-08-11 20:42:09 +00:00
would use too many resources, or if it can be better coordinated using a
single process.
## protocol overview
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
As usual, the protocol starts by the external special remote sending
the version of the protocol it's using.
VERSION 1
draft async extension
2020-08-11 20:42:09 +00:00
This extension is negotiated by git-annex sending an `EXTENSIONS` message
that includes `ASYNC`, and the external special remote responding in kind.
EXTENSIONS INFO ASYNC
EXTENSIONS ASYNC
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
async extension done
2020-08-14 19:24:34 +00:00
(Older versions of git-annex will not include `ASYNC` in their extensions
list. To support them, it's a good idea for the external special remote to
fall back to using the regular protocol.)
Once the extension is negotiated, messages in the protocol are
tagged with a job number, by prefixing them with "J n".
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
async extension done
2020-08-14 19:24:34 +00:00
As usual, the first message git-annex sends is generally PREPARE,
which gets tagged with a job number:
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
J 1 PREPARE
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
Rather than just responding PREPARE-SUCCESS, the job number has to be
included in the reply:
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
J 1 PREPARE-SUCCESS
draft async extension
2020-08-11 20:42:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
Suppose git-annex wants to make some transfers. It can request several
at the same time, using different job numbers:
draft async extension
2020-08-11 20:42:09 +00:00
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
J 1 TRANSFER RETRIEVE Key1 file1
J 2 TRANSFER RETRIEVE Key2 file2
draft async extension
2020-08-11 20:42:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
The special remote can now perform both transfers at the same time.
If it sends PROGRESS messages for these transfers, they have to be tagged
async extension done
2020-08-14 19:24:34 +00:00
with the job numbers:
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
J 1 PROGRESS 10
J 2 PROGRESS 500
J 1 PROGRESS 20
draft async extension
2020-08-11 20:42:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
The special remote can also send messages that query git-annex for some
information. These messages and the reply will also be tagged with a job
number.
draft async extension
2020-08-11 20:42:09 +00:00
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
J 1 GETCONFIG url
J 3 RETRIEVE Key3 file3
J 1 VALUE http://example.com/
draft async extension
2020-08-11 20:42:09 +00:00
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
One transfers are done, the special remote sends `TRANSFER-SUCCESS` tagged
with the job number.
rethought this protocol again Now that I've started implementation, I see it's really necessary that every message the special remote sends use the protocol, otherwise nasty edge cases abound.
2020-08-12 19:12:09 +00:00
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
J 2 TRANSFER-SUCCESS RETRIEVE Key2
J 1 PROGRESS 100
J 1 TRANSFER-SUCCESS RETRIEVE Key1
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
Lots of different jobs can be requested at the same time.
J 4 CHECKPRESENT Key3
J 5 CHECKPRESENT Key4
async extension done
2020-08-14 19:24:34 +00:00
J 6 REMOVE Key5
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
J 4 CHECKPRESENT-SUCCESS Key3
async extension done
2020-08-14 19:24:34 +00:00
J 6 REMOVE-SUCCESS Key5
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
J 5 CHECKPRESENT-FAILURE Key4
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
## notes
switch to TMVars for thread safety when using the async extension TVars were not updated atomically, which was ok when each thread got its own External that was the only thing using these TVars. But, with the async extension, several External instances can share the same var, so it needs to be a TMVar to avoid read/write conflicts. In particular, this makes PREPARE only be sent once.
2020-08-14 18:40:30 +00:00
There will be one job number for each thread that git-annex runs
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
concurrently, so around the same number as the -J value, although in some
cases git-annex does more concurrent operations than the -J value.
`PREPARE` is sent only once per run of a special remote
program, and despite being tagged with a job number, it should prepare the
special remote to run that and any other jobs.
rethought the async protocol some more Moving jobid generation to the git-annex side lets it be simplified a lot. Note that it will also be possible to generate one jobid per connection, rather than a new job per request. That will make overflow not an issue, and will avoid some work, and will simplify some of the code.
2020-08-14 00:18:06 +00:00
one jobid per thread And, relay ERROR on to all listening threads.
2020-08-14 18:24:46 +00:00
`ERROR` should not be tagged with a job number if either git-annex
or the special remote needs to send it.
async extension done
2020-08-14 19:24:34 +00:00
`VERSION`, `EXTENSIONS` and `ERROR` are the only protocol messages
that do not get tagged with a job number.
Reference in a new issue Copy permalink