Merge branch 'master' into httpproto

doc/design/p2p_protocol.mdwn

@@ -66,7 +66,7 @@ Now both client and server should use version 1.
 
 ## Cluster cycle prevention
 
-In protocol version 2, immediately after VERSION, the
+In protocol version 2 and above, immediately after VERSION, the
 client can send an additional message that is used to
 prevent cycles when accessing clusters.
 
@@ -135,9 +135,9 @@ The server responds with either SUCCESS or FAILURE.
 
 Note that if the content was not present, SUCCESS will be returned.
 
-In protocol version 2, the server can optionally reply with SUCCESS-PLUS
+In protocol version 2 and above, the server can optionally reply with
-or FAILURE-PLUS. Each has a subsequent list of UUIDs of repositories
+SUCCESS-PLUS or FAILURE-PLUS. Each has a subsequent list of UUIDs of
-that the content was removed from.
+repositories that the content was removed from.
 
 ## Removing content before a specified time
 

doc/design/p2p_protocol_over_http/draft1.mdwn

@@ -30,9 +30,7 @@ may want git-annex to use HTTP in eg a LAN.
 ## protocol version
 
 Each request in the protocol is versioned. The versions correspond
-to P2P protocol versions, but for simplicity, the minimum version supported
+to P2P protocol versions.
-over HTTP is version 3. Every implementation of the HTTP protocol must
-support version 3.
 
 The protocol version comes before the request. Eg: `/git-annex/v3/put`
 
@@ -65,6 +63,8 @@ Any request may also optionally include these parameters:
   This parameter can be given multiple times to list several cluster
   gateway UUIDs.
 
+  This parameter is only available for v3 and above.
+
 [Internally, git-annex can use these common parameters, plus the protocol
 version, to create a P2P session. The P2P session is driven through
 the AUTH, VERSION, and BYPASS messages, leaving the session ready to
@@ -79,9 +79,11 @@ It is not part of the P2P protocol per se, but is provided to let
 other clients than git-annex easily download the content of keys from the
 http server.
 
-This behaves the same as `GET /git-annex/v3/key/$key`, although its
+This behaves almost the same as `GET /git-annex/v3/key/$key`, although its
 behavior may change in later versions.
 
+When the key is not present on the server, this returns a 404 Not Found.
+
 ### GET /git-annex/v3/key/$key
 
 Get the content of a key from the server.
@@ -130,7 +132,21 @@ X-git-annex-data-length header indicated, then the data is invalid and
 should not be used. This can happen when eg, the data was being sent from
 an unlocked annexed file, which got modified while it was being sent.
 
-When the content is not present, the server will respond with 404.
+When the content is not present, the server will respond with 
+422 Unprocessable Content.
+
+### GET /git-annex/v2/key/$key
+
+Identical to v3.
+
+### GET /git-annex/v1/key/$key
+
+Identical to v3.
+
+### GET /git-annex/v0/key/$key
+
+Same as v3, except there is no X-git-annex-data-length header.
+Additional checking client-side will be required to validate the data.
 
 ### POST /git-annex/v3/checkpresent
 
@@ -148,6 +164,18 @@ The body of the request is empty.
 The server responds with a JSON object with a "present" field that is true
 if the key is present, or false if it is not present.
 
+### POST /git-annex/v2/checkpresent
+
+Identical to v3.
+
+### POST /git-annex/v1/checkpresent
+
+Identical to v3.
+
+### POST /git-annex/v0/checkpresent
+
+Identical to v3.
+
 ### POST /git-annex/v3/lockcontent
 
 Locks the content of a key on the server, preventing it from being removed.
@@ -172,6 +200,18 @@ If the client disconnects without sending "UNLOCKCONTENT", or the web
 server gets shut down before it can receive that, the content will remain
 locked for at least 10 minutes from when the server sent "SUCCESS".
 
+### POST /git-annex/v2/lockcontent
+
+Identical to v3.
+
+### POST /git-annex/v1/lockcontent
+
+Identical to v3.
+
+### POST /git-annex/v0/lockcontent
+
+Identical to v3.
+
 ### POST /git-annex/v3/remove
 
 Remove a key's content from the server.
@@ -196,6 +236,18 @@ If the server does not allow removing the key due to a policy
 (eg due to being read-only or append-only), it will respond with a JSON
 object with an "error" field that has an error message as its value.
 
+### POST /git-annex/v2/remove
+
+Identical to v3.
+
+### POST /git-annex/v1/remove
+
+Same as v3, except the JSON will not include "plusuuids".
+
+### POST /git-annex/v0/remove
+
+Identival to v1.
+
 ## POST /git-annex/v3/remove-before
 
 Remove a key's content from the server, but only before a specified time.
@@ -281,6 +333,19 @@ If the server does not allow storing the key due eg to a policy
 invalid, or because it ran out of disk space, it will respond with a
 JSON object with an "error" field that has an error message as its value.
 
+### POST /git-annex/v2/put
+
+Identical to v3.
+
+### POST /git-annex/v1/put
+
+Same as v3, except the JSON will not include "plusuuids".
+
+### POST /git-annex/v0/put
+
+Same as v1, except there is no X-git-annex-data-length header.
+Additional checking client-side will be required to validate the data.
+
 ### POST /git-annex/v3/putoffset
 
 Asks the server what `offset` can be used in a `put` of a key.
@@ -301,6 +366,12 @@ The body of the request is empty.
 The server responds with a JSON object with an "offset" field that 
 is the largest allowable offset.
 
+If the server already has the content of the key, it will respond with a
+JSON object with an "alreadyhave" field that is set to true. This JSON
+object may also have a field "plusuuids" that lists 
+the UUIDs of other repositories where the content is stored, in addition to
+the serveruuid.
+
 If the server does not allow storing the key due to a policy
 (eg due to being read-only or append-only), it will respond with a JSON
 object with an "error" field that has an error message as its value.
@@ -311,6 +382,14 @@ part way through a `PUT`, a synthetic empty `DATA` followed by `INVALID`
 will be used to get the P2P protocol back into a state where it will accept
 any request.]
 
+### POST /git-annex/v2/putoffset
+
+Identical to v3.
+
+### POST /git-annex/v1/putoffset
+
+Same as v3, except the JSON will not include "plusuuids".
+
 ## parts of P2P protocol that are not supported over HTTP
 
 `NOTIFYCHANGE` is not supported, but it would be possible to extend

doc/forum/Questions_about_borg_special_remotes/comment_3_a036be41663d81a79fe47e9021208169._comment

@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="git-annex@4a0625db6ced1ac00744697d5bac41393bcde646"
+ nickname="git-annex"
+ avatar="http://cdn.libravatar.org/avatar/d9fd8db33996d9a1da6b31045c43accf"
+ subject="comment 3"
+ date="2024-07-05T10:22:45Z"
+ content="""
+Is there a way to configure a certain BORG_PASSCOMMAND to always be used for a given remote. In general, is there a way to set environment variables per remote?
+"""]]

doc/todo/git-annex_proxies.mdwn

@@ -28,19 +28,20 @@ Planned schedule of work:
 
 ## work notes
 
-* websockets or something else for LOCKCONTENT over http?
+* Next step: Ready to begin implementing in servant. I have a file
+  `servant.hs` in the httpproto branch that works through some of the
+  bytestring streaming issues.
 
-* Should integrate into an existing web server similarly to git
+* Perhaps: Support cgi program that proxies over to a webserver
-  smart http server. May as well also support it as a standalone
+  speaking the http protocol.
-  webserver.
-
-* endpoint versioning should include v1 and v0 to support connections
-  proxied from older clients
 
 ## completed items for July's work on p2p protocol over http
 
 * addressed [[doc/todo/P2P_locking_connection_drop_safety]]
 
+* finalized HTTP P2P protocol draft 1,
+  [[design/p2p_protocol_over_http/draft1]]
+
 ## items deferred until later for [[design/passthrough_proxy]]
 
 * Check annex.diskreserve when proxying for special remotes