diff --git a/CHANGELOG b/CHANGELOG index 75ed3b5153..5c1b4aa3c3 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,4 +1,13 @@ -git-annex (8.20200810) upstream; urgency=medium +git-annex (8.20200815) UNRELEASED; urgency=medium + + * The external special remote protocol got an ASYNC extension. + This can be used by an external special remote to let a single process + perform concurrent actions, rather than multiple processes being + started, when that is more efficient. + + -- Joey Hess Fri, 14 Aug 2020 14:57:45 -0400 + +git-annex (8.20200814) upstream; urgency=medium * Added support for external backend programs. So if you want a hash that git-annex doesn't support, or something stranger, you can write a diff --git a/doc/design/external_special_remote_protocol/async_appendix.mdwn b/doc/design/external_special_remote_protocol/async_appendix.mdwn index e9e44452d3..acebfdb25b 100644 --- a/doc/design/external_special_remote_protocol/async_appendix.mdwn +++ b/doc/design/external_special_remote_protocol/async_appendix.mdwn @@ -1,5 +1,3 @@ -(This is a draft and not implemented yet.) - This is an appendix to the [[external_special_remote_protocol]]. [[!toc]] @@ -10,7 +8,7 @@ Normally, an external special remote can only be used to do one thing at a time, and when git-annex has concurrency enabled, it will start up multiple processes for the same external special remote. -This extension lets a single external special remote process handle +The `ASYNC` extension lets a single external special remote process handle multiple concurrent jobs, which can be useful if multiple processes would use too many resources, or if it can be better coordinated using a single process. @@ -28,10 +26,15 @@ that includes `ASYNC`, and the external special remote responding in kind. EXTENSIONS INFO ASYNC EXTENSIONS ASYNC -From this point forward, every message in the protocol is tagged with a job -number, by prefixing it with "J n". +(Older versions of git-annex will not include `ASYNC` in their extensions +list. To support them, it's a good idea for the external special remote to +fall back to using the regular protocol.) -As usual, the first message git-annex sends is generally PREPARE: +Once the extension is negotiated, messages in the protocol are +tagged with a job number, by prefixing them with "J n". + +As usual, the first message git-annex sends is generally PREPARE, +which gets tagged with a job number: J 1 PREPARE @@ -48,7 +51,7 @@ at the same time, using different job numbers: The special remote can now perform both transfers at the same time. If it sends PROGRESS messages for these transfers, they have to be tagged -with the job number: +with the job numbers: J 1 PROGRESS 10 J 2 PROGRESS 500 @@ -73,18 +76,11 @@ Lots of different jobs can be requested at the same time. J 4 CHECKPRESENT Key3 J 5 CHECKPRESENT Key4 - J 6 REMOVE Key3 + J 6 REMOVE Key5 J 4 CHECKPRESENT-SUCCESS Key3 - J 6 REMOVE-SUCCESS Key3 + J 6 REMOVE-SUCCESS Key5 J 5 CHECKPRESENT-FAILURE Key4 -An example of sending multiple replies to a request is `LISTCONFIGS`, eg: - - J 7 LISTCONFIGS - J 7 CONFIG foo some config - J 7 CONFIG bar other config - J 7 CONFIGEND - ## notes There will be one job number for each thread that git-annex runs @@ -97,3 +93,6 @@ special remote to run that and any other jobs. `ERROR` should not be tagged with a job number if either git-annex or the special remote needs to send it. + +`VERSION`, `EXTENSIONS` and `ERROR` are the only protocol messages +that do not get tagged with a job number. diff --git a/doc/devblog/day_629__async_external_special_remotes.mdwn b/doc/devblog/day_629__async_external_special_remotes.mdwn new file mode 100644 index 0000000000..eefc8958be --- /dev/null +++ b/doc/devblog/day_629__async_external_special_remotes.mdwn @@ -0,0 +1,25 @@ +After a release on Monday, I've spent the week working on +[[async extension to external special remote protocol|design/external_special_remote_protocol/async_appendix]]. +This is lets a single external special remote process handle multiple +requests at the same time, when it's more efficient to use one process +than for git-annex to run several processes. + +It's a good thing I added support for extensions a couple of years back. +I never imagined at the time using it for something like this, that +radically changes the whole protocol! It could have just been protocol +version 2, but then special remotes would be pushed towards using this by +default, which I don't want. It's probably overkill for most of them. + + J 4 CHECKPRESENT Key3 + J 5 CHECKPRESENT Key4 + J 6 REMOVE Key5 + J 4 CHECKPRESENT-SUCCESS Key3 + J 6 REMOVE-SUCCESS Key5 + J 5 CHECKPRESENT-FAILURE Key4 + +The protocol extension went through a bunch of iterations, ending up with +probably the simplest possible way to do it, a simple framing layer around +the main protocol. I started with rather a lot of rather hairy code and it +kind of all melted away as I refined the protocol down to that, which was +nice, although I also kind of wish I had been able to jump right to +the clean and simple end result. diff --git a/doc/todo/idea__58___external_special_remote___34__async__34___protocol_for_transfers.mdwn b/doc/todo/idea__58___external_special_remote___34__async__34___protocol_for_transfers.mdwn index d9fb7f2e28..3fd55dede1 100644 --- a/doc/todo/idea__58___external_special_remote___34__async__34___protocol_for_transfers.mdwn +++ b/doc/todo/idea__58___external_special_remote___34__async__34___protocol_for_transfers.mdwn @@ -8,3 +8,5 @@ Just an idea ;) [[!meta author=yoh]] [[!tag projects/dandi]] + +> [[done]] --[[Joey]] diff --git a/doc/todo/idea__58___external_special_remote___34__async__34___protocol_for_transfers/comment_7_c0f5b5098252df0f86528078f3c96ec7._comment b/doc/todo/idea__58___external_special_remote___34__async__34___protocol_for_transfers/comment_7_c0f5b5098252df0f86528078f3c96ec7._comment new file mode 100644 index 0000000000..079368ed48 --- /dev/null +++ b/doc/todo/idea__58___external_special_remote___34__async__34___protocol_for_transfers/comment_7_c0f5b5098252df0f86528078f3c96ec7._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 7""" + date="2020-08-14T18:59:37Z" + content=""" +ASYNC extension is implemented. The protocol went through several +iterations and ended up at about the simplest and cleanest possible way to +do it. +"""]]