async extension done

CHANGELOG

@@ -1,4 +1,13 @@
-git-annex (8.20200810) upstream; urgency=medium
+git-annex (8.20200815) UNRELEASED; urgency=medium
+
+  * The external special remote protocol got an ASYNC extension.
+    This can be used by an external special remote to let a single process
+    perform concurrent actions, rather than multiple processes being
+    started, when that is more efficient.
+
+ -- Joey Hess <id@joeyh.name>  Fri, 14 Aug 2020 14:57:45 -0400
+
+git-annex (8.20200814) upstream; urgency=medium
 
   * Added support for external backend programs. So if you want a hash
     that git-annex doesn't support, or something stranger, you can write a

doc/design/external_special_remote_protocol/async_appendix.mdwn

@@ -1,5 +1,3 @@
-(This is a draft and not implemented yet.)
-
 This is an appendix to the [[external_special_remote_protocol]].
 
 [[!toc]]
@@ -10,7 +8,7 @@ Normally, an external special remote can only be used to do one thing at a
 time, and when git-annex has concurrency enabled, it will start up multiple
 processes for the same external special remote. 
 
-This extension lets a single external special remote process handle
+The `ASYNC` extension lets a single external special remote process handle
 multiple concurrent jobs, which can be useful if multiple processes
 would use too many resources, or if it can be better coordinated using a
 single process.
@@ -28,10 +26,15 @@ that includes `ASYNC`, and the external special remote responding in kind.
 	EXTENSIONS INFO ASYNC
 	EXTENSIONS ASYNC
 
-From this point forward, every message in the protocol is tagged with a job
-number, by prefixing it with "J n".
+(Older versions of git-annex will not include `ASYNC` in their extensions
+list. To support them, it's a good idea for the external special remote to
+fall back to using the regular protocol.)
 
-As usual, the first message git-annex sends is generally PREPARE:
+Once the extension is negotiated, messages in the protocol are
+tagged with a job number, by prefixing them with "J n".
+
+As usual, the first message git-annex sends is generally PREPARE,
+which gets tagged with a job number:
 
 	J 1 PREPARE
 
@@ -48,7 +51,7 @@ at the same time, using different job numbers:
 
 The special remote can now perform both transfers at the same time.
 If it sends PROGRESS messages for these transfers, they have to be tagged
-with the job number:
+with the job numbers:
 	
 	J 1 PROGRESS 10
 	J 2 PROGRESS 500
@@ -73,18 +76,11 @@ Lots of different jobs can be requested at the same time.
 
 	J 4 CHECKPRESENT Key3
 	J 5 CHECKPRESENT Key4
-	J 6 REMOVE Key3
+	J 6 REMOVE Key5
 	J 4 CHECKPRESENT-SUCCESS Key3
-	J 6 REMOVE-SUCCESS Key3
+	J 6 REMOVE-SUCCESS Key5
 	J 5 CHECKPRESENT-FAILURE Key4
 
-An example of sending multiple replies to a request is `LISTCONFIGS`, eg:
-
-	J 7 LISTCONFIGS
-	J 7 CONFIG foo some config
-	J 7 CONFIG bar other config
-	J 7 CONFIGEND
-
 ## notes
 
 There will be one job number for each thread that git-annex runs
@@ -97,3 +93,6 @@ special remote to run that and any other jobs.
 
 `ERROR` should not be tagged with a job number if either git-annex
 or the special remote needs to send it.
+
+`VERSION`, `EXTENSIONS` and `ERROR` are the only protocol messages
+that do not get tagged with a job number.

doc/devblog/day_629__async_external_special_remotes.mdwn

@@ -0,0 +1,25 @@
+After a release on Monday, I've spent the week working on
+[[async extension to external special remote protocol|design/external_special_remote_protocol/async_appendix]].
+This is lets a single external special remote process handle multiple
+requests at the same time, when it's more efficient to use one process
+than for git-annex to run several processes.
+
+It's a good thing I added support for extensions a couple of years back.
+I never imagined at the time using it for something like this, that
+radically changes the whole protocol! It could have just been protocol
+version 2, but then special remotes would be pushed towards using this by
+default, which I don't want. It's probably overkill for most of them.
+
+          J 4 CHECKPRESENT Key3
+          J 5 CHECKPRESENT Key4
+          J 6 REMOVE Key5
+          J 4 CHECKPRESENT-SUCCESS Key3
+          J 6 REMOVE-SUCCESS Key5
+          J 5 CHECKPRESENT-FAILURE Key4
+
+The protocol extension went through a bunch of iterations, ending up with
+probably the simplest possible way to do it, a simple framing layer around
+the main protocol. I started with rather a lot of rather hairy code and it
+kind of all melted away as I refined the protocol down to that, which was
+nice, although I also kind of wish I had been able to jump right to
+the clean and simple end result.

doc/todo/idea__58___external_special_remote___34__async__34___protocol_for_transfers.mdwn

@@ -8,3 +8,5 @@ Just an idea ;)
 
 [[!meta author=yoh]]
 [[!tag projects/dandi]]
+
+> [[done]] --[[Joey]] 

doc/todo/idea__58___external_special_remote___34__async__34___protocol_for_transfers/comment_7_c0f5b5098252df0f86528078f3c96ec7._comment

@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 7"""
+ date="2020-08-14T18:59:37Z"
+ content="""
+ASYNC extension is implemented. The protocol went through several
+iterations and ended up at about the simplest and cleanest possible way to
+do it.
+"""]]