From d782b136e0ec5afa411fb901fd7afc5db7135412 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Mon, 28 Oct 2024 14:16:15 -0400 Subject: [PATCH] p2p protocol version 4 for DATA-PRESENT --- doc/design/p2p_protocol.mdwn | 19 ++++- doc/design/p2p_protocol_over_http.mdwn | 107 ++++++++++++++++++------- 2 files changed, 92 insertions(+), 34 deletions(-) diff --git a/doc/design/p2p_protocol.mdwn b/doc/design/p2p_protocol.mdwn index 5b15142a52..d9cf75a02f 100644 --- a/doc/design/p2p_protocol.mdwn +++ b/doc/design/p2p_protocol.mdwn @@ -56,7 +56,7 @@ any authentication. The client sends the highest protocol version it supports: - VERSION 3 + VERSION 4 The server responds with the highest protocol version it supports that is less than or equal to the version the client sent: @@ -190,7 +190,8 @@ ALREADY-HAVE-PLUS. The subsequent list of UUIDs are additional UUIDs where the content is stored, in addition to the UUID where the client was going to send it. -Otherwise, it responds with: +When the server wants the client to send the content to it, +it replies with: PUT-FROM Offset @@ -200,7 +201,7 @@ the client to start. This allows resuming transfers. The client then sends a DATA message with content of the file from the offset to the end of file. -In protocol version 1 and above, after the data, the client sends an +In protocol version 1 and above, after the DATA, the client sends an additional message, to indicate if the content of the file has changed while it was being sent. @@ -213,6 +214,18 @@ it replies with SUCCESS. Otherwise, FAILURE. In protocol version 2 and above, the server can optionally reply with SUCCESS-PLUS and a list of UUIDs where the content was stored. +In protocol version 4 and above, rather than sending a DATA message, +the client can send the content to the repository in some other way +than using the protocol. To indicate that it has sent the content +like that, the client sends: + + DATA-PRESENT + +The server responds to DATA-PRESENT by checking if the content is +present in the repository. If so it proceeds the same as if the client +had used DATA to send the data and then sent VALID. Eg, it responds with +SUCCESS (or FAILURE if it cannot verify that the data is present). + ## Getting content from the server To get content from the server, the client sends: diff --git a/doc/design/p2p_protocol_over_http.mdwn b/doc/design/p2p_protocol_over_http.mdwn index 1c5ff28a61..0e49782fa0 100644 --- a/doc/design/p2p_protocol_over_http.mdwn +++ b/doc/design/p2p_protocol_over_http.mdwn @@ -49,7 +49,7 @@ over HTTPS. Requests are versioned. The versions correspond to P2P protocol versions. The version is part of the request path, -eg "v3" +eg "v4" If the server does not support a particular version, the request will fail with a 404, and the client should fall @@ -99,7 +99,7 @@ with 404 Not Found. Note that the common parameters bypass and clientuuid, while accepted, have no effect. Both are optional for this request. -### GET /git-annex/$uuid/v3/key/$key +### GET /git-annex/$uuid/v4/key/$key Get the content of a key from the repository with the specified uuid. @@ -144,20 +144,24 @@ an unlocked annexed file, which got modified while it was being sent. When the key is not present on the server, it will respond with 404 Not Found. +### GET /git-annex/$uuid/v3/key/$key + +Identical to v4. + ### GET /git-annex/$uuid/v2/key/$key -Identical to v3. +Identical to v4. ### GET /git-annex/$uuid/v1/key/$key -Identical to v3. +Identical to v4. ### GET /git-annex/$uuid/v0/key/$key -Same as v3, except the X-git-annex-data-length header is not used. +Same as v4, except the X-git-annex-data-length header is not used. Additional checking client-side will be required to validate the data. -### POST /git-annex/$uuid/v3/checkpresent +### POST /git-annex/$uuid/v4/checkpresent Checks if a key is currently present on the server. @@ -173,19 +177,23 @@ The body of the request is empty. The server responds with a JSON object with a "present" field that is true if the key is present, or false if it is not present. +### POST /git-annex/$uuid/v3/checkpresent + +Identical to v4. + ### POST /git-annex/$uuid/v2/checkpresent -Identical to v3. +Identical to v4. ### POST /git-annex/$uuid/v1/checkpresent -Identical to v3. +Identical to v4. ### POST /git-annex/$uuid/v0/checkpresent -Identical to v3. +Identical to v4. -### POST /git-annex/$uuid/v3/lockcontent +### POST /git-annex/$uuid/v4/lockcontent Locks the content of a key on the server, preventing it from being removed. @@ -203,19 +211,23 @@ The key will remain locked for 10 minutes. But, usually `keeplocked` is used to control the lifetime of the lock, using the "lockid" parameter from the server's reply. (See below.) +### POST /git-annex/$uuid/v3/lockcontent + +Identical to v4. + ### POST /git-annex/$uuid/v2/lockcontent -Identical to v3. +Identical to v4. ### POST /git-annex/$uuid/v1/lockcontent -Identical to v3. +Identical to v4. ### POST /git-annex/$uuid/v0/lockcontent -Identical to v3. +Identical to v4. -### POST /git-annex/$uuid/v3/keeplocked +### POST /git-annex/$uuid/v4/keeplocked Controls the lifetime of a lock on a key that was earlier obtained with `lockcontent`. @@ -252,21 +264,25 @@ locked for 10 minutes from the time it was first locked. Note that the common parameters bypass and clientuuid, while accepted, have no effect. +### POST /git-annex/$uuid/v3/keeplocked + +Identical to v4. + ### POST /git-annex/$uuid/v2/keeplocked -Identical to v3. +Identical to v4. ### POST /git-annex/$uuid/v1/keeplocked -Identical to v3. +Identical to v4. ### POST /git-annex/$uuid/v0/keeplocked -Identical to v3. +Identical to v4. -### POST /git-annex/$uuid/v3/remove +### POST /git-annex/$uuid/v4/remove -Remove a key's content from the server. +Remove a key's content from the repository. Example: @@ -284,21 +300,25 @@ or false if the key was not able to be removed. The JSON object can have an additional field "plusuuids" that is a list of UUIDs of other repositories that the content was removed from. +### POST /git-annex/$uuid/v3/remove + +Identical to v4. + ### POST /git-annex/$uuid/v2/remove -Identical to v3. +Identical to v4. ### POST /git-annex/$uuid/v1/remove -Same as v3, except the JSON will not include "plusuuids". +Same as v4, except the JSON will not include "plusuuids". ### POST /git-annex/$uuid/v0/remove Identical to v1. -### POST /git-annex/$uuid/v3/remove-before +### POST /git-annex/$uuid/v4/remove-before -Remove a key's content from the server, but only before a specified time. +Remove a key's content from the repository, but only before a specified time. Example: @@ -314,7 +334,11 @@ removal will fail and the server will respond with: `{"removed": false}` This is used to avoid removing content after a point in time where it is no longer locked in other repostitories. -### POST /git-annex/$uuid/v3/gettimestamp +### POST /git-annex/$uuid/v3/remove-before + +Identical to v4. + +### POST /git-annex/$uuid/v4/gettimestamp Gets the current timestamp from the server. @@ -331,9 +355,13 @@ current value of its monotonic clock, as a number of seconds. Important: If multiple servers are serving this API for the same repository, they MUST all use the same monotonic clock. -### POST /git-annex/$uuid/v3/put +### POST /git-annex/$uuid/v3/gettimestamp -Store content on the server. +Identical to v4. + +### POST /git-annex/$uuid/v4/put + +Store content in the repository. Example: @@ -358,11 +386,18 @@ There are are also these optional parameters: Number of bytes that have been omitted from the beginning of the file. Usually this will be determined by making a `putoffset` request. +* `data-present` + + When set to 1, this indicates that the data has been sent to the repository + in some other way. The body of the request will be empty. The server will + verify that the data is present in the repository and will proceed the + same as if the data was sent in the request. + The `Content-Type` header should be `application/octet-stream`. The `X-git-annex-data-length` must be included. It indicates the number -of bytes of content that are expected to be sent. -Note that there is no need to send a Content-Length header. +of bytes of content that are expected to be sent in the body of the +request. Note that there is no need to send a Content-Length header. If the length of the body is different than what the the X-git-annex-data-length header indicated, then the data is invalid and @@ -371,10 +406,16 @@ an unlocked annexed file, which got modified while it was being sent. The server responds with a JSON object with a field "stored" that is true if it received the data and stored the content. +(Or when `data-out-of-band` is used, if it has verified that the key is +present in the repository.) The JSON object can have an additional field "plusuuids" that is a list of UUIDs of other repositories that the content was stored to. +### POST /git-annex/$uuid/v3/put + +Identical to v4 except the `data-out-of-band` parameter cannot be used. + ### POST /git-annex/$uuid/v2/put Identical to v3. @@ -387,7 +428,7 @@ Same as v3, except the JSON will not include "plusuuids". Same as v1, except additional checking is done to validate the data. -### POST /git-annex/$uuid/v3/putoffset +### POST /git-annex/$uuid/v4/putoffset Asks the server what `offset` can be used in a `put` of a key. @@ -419,13 +460,17 @@ part way through a `PUT`, a synthetic empty `DATA` followed by `INVALID` will be used to get the P2P protocol back into a state where it will accept any request.] +### POST /git-annex/$uuid/v3/putoffset + +Identical to v4. + ### POST /git-annex/$uuid/v2/putoffset -Identical to v3. +Identical to v4. ### POST /git-annex/$uuid/v1/putoffset -Same as v3, except the JSON will not include "plusuuids". +Same as v4, except the JSON will not include "plusuuids". ## parts of P2P protocol that are not supported over HTTP