From 40306d3fcf6774b8653a5ab09ce42ba47eda3083 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 5 Jul 2024 15:34:58 -0400 Subject: [PATCH] finalizing HTTP P2p protocol some more Added v2-v0 endpoints. These are tedious, but will be needed in order to use the HTTP protocol to proxy to repositories with older git-annex, where git-annex-shell will be speaking an older version of the protocol. Changed GET to use 422 when the content is not present. 404 is needed to detect when a protocol version is not supported. --- doc/design/p2p_protocol.mdwn | 8 +- doc/design/p2p_protocol_over_http/draft1.mdwn | 89 +++++++++++++++++-- doc/todo/git-annex_proxies.mdwn | 15 ++-- 3 files changed, 96 insertions(+), 16 deletions(-) diff --git a/doc/design/p2p_protocol.mdwn b/doc/design/p2p_protocol.mdwn index c4f4aac27b..d886376a15 100644 --- a/doc/design/p2p_protocol.mdwn +++ b/doc/design/p2p_protocol.mdwn @@ -66,7 +66,7 @@ Now both client and server should use version 1. ## Cluster cycle prevention -In protocol version 2, immediately after VERSION, the +In protocol version 2 and above, immediately after VERSION, the client can send an additional message that is used to prevent cycles when accessing clusters. @@ -135,9 +135,9 @@ The server responds with either SUCCESS or FAILURE. Note that if the content was not present, SUCCESS will be returned. -In protocol version 2, the server can optionally reply with SUCCESS-PLUS -or FAILURE-PLUS. Each has a subsequent list of UUIDs of repositories -that the content was removed from. +In protocol version 2 and above, the server can optionally reply with +SUCCESS-PLUS or FAILURE-PLUS. Each has a subsequent list of UUIDs of +repositories that the content was removed from. ## Removing content before a specified time diff --git a/doc/design/p2p_protocol_over_http/draft1.mdwn b/doc/design/p2p_protocol_over_http/draft1.mdwn index 656fc08cee..a776caafe6 100644 --- a/doc/design/p2p_protocol_over_http/draft1.mdwn +++ b/doc/design/p2p_protocol_over_http/draft1.mdwn @@ -18,9 +18,7 @@ may want git-annex to use HTTP in eg a LAN. ## protocol version Each request in the protocol is versioned. The versions correspond -to P2P protocol versions, but for simplicity, the minimum version supported -over HTTP is version 3. Every implementation of the HTTP protocol must -support version 3. +to P2P protocol versions. The protocol version comes before the request. Eg: `/git-annex/v3/put` @@ -53,6 +51,8 @@ Any request may also optionally include these parameters: This parameter can be given multiple times to list several cluster gateway UUIDs. + This parameter is only available for v3 and above. + [Internally, git-annex can use these common parameters, plus the protocol version, to create a P2P session. The P2P session is driven through the AUTH, VERSION, and BYPASS messages, leaving the session ready to @@ -67,9 +67,11 @@ It is not part of the P2P protocol per se, but is provided to let other clients than git-annex easily download the content of keys from the http server. -This behaves the same as `GET /git-annex/v3/key/$key`, although its +This behaves almost the same as `GET /git-annex/v3/key/$key`, although its behavior may change in later versions. +When the key is not present on the server, this returns a 404 Not Found. + ### GET /git-annex/v3/key/$key Get the content of a key from the server. @@ -118,7 +120,21 @@ X-git-annex-data-length header indicated, then the data is invalid and should not be used. This can happen when eg, the data was being sent from an unlocked annexed file, which got modified while it was being sent. -When the content is not present, the server will respond with 404. +When the content is not present, the server will respond with +422 Unprocessable Content. + +### GET /git-annex/v2/key/$key + +Identical to v3. + +### GET /git-annex/v1/key/$key + +Identical to v3. + +### GET /git-annex/v0/key/$key + +Same as v3, except there is no X-git-annex-data-length header. +Additional checking client-side will be required to validate the data. ### POST /git-annex/v3/checkpresent @@ -136,6 +152,18 @@ The body of the request is empty. The server responds with a JSON object with a "present" field that is true if the key is present, or false if it is not present. +### POST /git-annex/v2/checkpresent + +Identical to v3. + +### POST /git-annex/v1/checkpresent + +Identical to v3. + +### POST /git-annex/v0/checkpresent + +Identical to v3. + ### POST /git-annex/v3/lockcontent Locks the content of a key on the server, preventing it from being removed. @@ -160,6 +188,18 @@ If the client disconnects without sending "UNLOCKCONTENT", or the web server gets shut down before it can receive that, the content will remain locked for at least 10 minutes from when the server sent "SUCCESS". +### POST /git-annex/v2/lockcontent + +Identical to v3. + +### POST /git-annex/v1/lockcontent + +Identical to v3. + +### POST /git-annex/v0/lockcontent + +Identical to v3. + ### POST /git-annex/v3/remove Remove a key's content from the server. @@ -184,6 +224,18 @@ If the server does not allow removing the key due to a policy (eg due to being read-only or append-only), it will respond with a JSON object with an "error" field that has an error message as its value. +### POST /git-annex/v2/remove + +Identical to v3. + +### POST /git-annex/v1/remove + +Same as v3, except the JSON will not include "plusuuids". + +### POST /git-annex/v0/remove + +Identival to v1. + ## POST /git-annex/v3/remove-before Remove a key's content from the server, but only before a specified time. @@ -269,6 +321,19 @@ If the server does not allow storing the key due eg to a policy invalid, or because it ran out of disk space, it will respond with a JSON object with an "error" field that has an error message as its value. +### POST /git-annex/v2/put + +Identical to v3. + +### POST /git-annex/v1/put + +Same as v3, except the JSON will not include "plusuuids". + +### POST /git-annex/v0/put + +Same as v1, except there is no X-git-annex-data-length header. +Additional checking client-side will be required to validate the data. + ### POST /git-annex/v3/putoffset Asks the server what `offset` can be used in a `put` of a key. @@ -289,6 +354,12 @@ The body of the request is empty. The server responds with a JSON object with an "offset" field that is the largest allowable offset. +If the server already has the content of the key, it will respond with a +JSON object with an "alreadyhave" field that is set to true. This JSON +object may also have a field "plusuuids" that lists +the UUIDs of other repositories where the content is stored, in addition to +the serveruuid. + If the server does not allow storing the key due to a policy (eg due to being read-only or append-only), it will respond with a JSON object with an "error" field that has an error message as its value. @@ -299,6 +370,14 @@ part way through a `PUT`, a synthetic empty `DATA` followed by `INVALID` will be used to get the P2P protocol back into a state where it will accept any request.] +### POST /git-annex/v2/putoffset + +Identical to v3. + +### POST /git-annex/v1/putoffset + +Same as v3, except the JSON will not include "plusuuids". + ## parts of P2P protocol that are not supported over HTTP `NOTIFYCHANGE` is not supported, but it would be possible to extend diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn index 841fbe13ce..58bb319562 100644 --- a/doc/todo/git-annex_proxies.mdwn +++ b/doc/todo/git-annex_proxies.mdwn @@ -28,19 +28,20 @@ Planned schedule of work: ## work notes -* websockets or something else for LOCKCONTENT over http? +* Next step: Ready to begin implementing in servant. I have a file + `servant.hs` in the httpproto branch that works through some of the + bytestring streaming issues. -* Should integrate into an existing web server similarly to git - smart http server. May as well also support it as a standalone - webserver. - -* endpoint versioning should include v1 and v0 to support connections - proxied from older clients +* Perhaps: Support cgi program that proxies over to a webserver + speaking the http protocol. ## completed items for July's work on p2p protocol over http * addressed [[doc/todo/P2P_locking_connection_drop_safety]] +* finalized HTTP P2P protocol draft 1, + [[design/p2p_protocol_over_http/draft1]] + ## items deferred until later for [[design/passthrough_proxy]] * Check annex.diskreserve when proxying for special remotes