added retrieveKeyFileInOrder and ORDERED to external special remote protocol

I anticipate lots of external special remote programs will neglect
implementing this. Still, it's the right thing to do to assume that some
of them may write files out of order. Probably most external special
remotes will not be used with a proxy. When someone is using one with a
proxy, they can always get it fixed to send ORDERED.

doc/design/external_special_remote_protocol.mdwn

@@ -55,8 +55,8 @@ any extensions it wants to use.
 
 Next, git-annex will generally send a message telling the special
 remote to start up. (Or it might send an INITREMOTE or EXPORTSUPPORTED or
-LISTCONFIGS, or perhaps other things in the future, so don't hardcode this
-order.)
+LISTCONFIGS, or perhaps other things in the future, so don't
+hardcode this order.)
 
 	PREPARE
 
@@ -118,7 +118,7 @@ The following requests *must* all be supported by the special remote.
 * `PREPARE`  
   Tells the remote that it's time to prepare itself to be used.  
   Only a few requests for details about the remote can come before this
-  (EXTENSIONS, INITREMOTE, EXPORTSUPPORTED, and LISTCONFIGS,
+  (EXTENSIONS, INITREMOTE, EXPORTSUPPORTED and LISTCONFIGS,
   but others may be added later).
   * `PREPARE-SUCCESS`  
     Sent as a response to PREPARE once the special remote is ready for use.
@@ -203,6 +203,15 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
   (See Config/Cost.hs for some standard costs.)
   * `COST Int`  
     Indicates the cost of the remote.
+* `GETORDERED`  
+  Asks the remote if it will always write files in order when performing a
+  `TRANSFER RETRIEVE`. Writing in order lets a proxy stream content from
+  the remote. When this is not implemented, git-annex assumes the
+  remote may write parts of the file out of order.
+  * `ORDERED`
+    Indicates that files are written in order.
+  * `UNORDERED`
+    Indicates that files are not written in order.
 * `GETAVAILABILITY`  
   Asks the remote if it is locally or globally available.
   (Ie stored in the cloud vs on a local disk.)  
@@ -221,6 +230,8 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
     trying to use the remote.  
     Older versions of git-annex do not support this response, so avoid
     sending it unless the `UNAVAILABLERESPONSE` extension is enabled.
+* `ORDERED`
+  
 * `CLAIMURL Url`  
   Asks the remote if it wishes to claim responsibility for downloading
   an url.

doc/design/passthrough_proxy.mdwn

@@ -364,10 +364,6 @@ remote to the usual temp object file on the proxy, but without moving that
 to the annex object file at the end. As the temp object file grows, stream
 the content out via the proxy. 
 
-> This needs the same process to read and write the same file, which is
-> disallowed in Haskell (without going lowlevel in a way that seems
-> difficult).
-
 Some special remotes will overwrite or truncate an existing temp object
 file when starting a download. So the proxy should wait until the file is
 growing to start streaming it.

doc/git-annex-updateproxy.mdwn

@@ -36,6 +36,9 @@ When that branch is pushed to the proxy, it will update the tree exported
 to the special remote. When files are copied to the remote via the proxy,
 it will also update the exported tree.
 
+Note that proxying for a special remote can temporarily use disk space
+in the repository, to temporarily hold the content of annexed files.
+
 # OPTIONS
 
 * The [[git-annex-common-options]](1) can be used.

doc/special_remotes/external/example.sh

@@ -222,6 +222,12 @@ while read line; do
 				;;
 			esac
 		;;
+		GETORDERED)
+			# This remote writes to files in order when
+			# retrieving them. If it didn't, it
+			# would be important to respond with UNORDERED.
+			echo ORDERED
+		;;
 		CHECKPRESENT)
 			key="$2"
 			calclocation "$key"

doc/todo/git-annex_proxies.mdwn

@@ -30,6 +30,12 @@ Planned schedule of work:
 
 * Currently working on streaming download via proxy from special remote.
 
+## completed items for October's work on streaming through proxy to special remotes
+
+* Stream downloads through proxy for all special remotes that indicate
+  they download in order.
+* Added ORDERED message to external special remote protocol.
+
 ## completed items for September's work on proving behavior of preferred content
 
 * Static analysis to detect "not present", "not balanced", and similar