mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

p2p protocol version 4 for DATA-PRESENT

Browse source
This commit is contained in:
Joey Hess 2024-10-28 14:16:15 -04:00
parent dc7aec77a4
commit d782b136e0
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 92 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

19
doc/design/p2p_protocol.mdwn
View file

@ -56,7 +56,7 @@ any authentication.
The client sends the highest protocol version it supports: The client sends the highest protocol version it supports:
VERSION 3 VERSION 4
The server responds with the highest protocol version it supports The server responds with the highest protocol version it supports
that is less than or equal to the version the client sent: that is less than or equal to the version the client sent:
@ -190,7 +190,8 @@ ALREADY-HAVE-PLUS. The subsequent list of UUIDs are additional
UUIDs where the content is stored, in addition to the UUID where UUIDs where the content is stored, in addition to the UUID where
the client was going to send it. the client was going to send it.
Otherwise, it responds with: When the server wants the client to send the content to it,
it replies with:
PUT-FROM Offset PUT-FROM Offset
@ -200,7 +201,7 @@ the client to start. This allows resuming transfers.
The client then sends a DATA message with content of the file from The client then sends a DATA message with content of the file from
the offset to the end of file. the offset to the end of file.
In protocol version 1 and above, after the data, the client sends an In protocol version 1 and above, after the DATA, the client sends an
additional message, to indicate if the content of the file has changed additional message, to indicate if the content of the file has changed
while it was being sent. while it was being sent.
@ -213,6 +214,18 @@ it replies with SUCCESS. Otherwise, FAILURE.
In protocol version 2 and above, the server can optionally reply with In protocol version 2 and above, the server can optionally reply with
SUCCESS-PLUS and a list of UUIDs where the content was stored. SUCCESS-PLUS and a list of UUIDs where the content was stored.
In protocol version 4 and above, rather than sending a DATA message,
the client can send the content to the repository in some other way
than using the protocol. To indicate that it has sent the content
like that, the client sends:
DATA-PRESENT
The server responds to DATA-PRESENT by checking if the content is
present in the repository. If so it proceeds the same as if the client
had used DATA to send the data and then sent VALID. Eg, it responds with
SUCCESS (or FAILURE if it cannot verify that the data is present).
## Getting content from the server ## Getting content from the server
To get content from the server, the client sends: To get content from the server, the client sends:

107
doc/design/p2p_protocol_over_http.mdwn
View file

@ -49,7 +49,7 @@ over HTTPS.
Requests are versioned. The versions correspond to Requests are versioned. The versions correspond to
P2P protocol versions. The version is part of the request path, P2P protocol versions. The version is part of the request path,
eg "v3" eg "v4"
If the server does not support a particular version, the If the server does not support a particular version, the
request will fail with a 404, and the client should fall request will fail with a 404, and the client should fall
@ -99,7 +99,7 @@ with 404 Not Found.
Note that the common parameters bypass and clientuuid, while Note that the common parameters bypass and clientuuid, while
accepted, have no effect. Both are optional for this request. accepted, have no effect. Both are optional for this request.
### GET /git-annex/$uuid/v3/key/$key ### GET /git-annex/$uuid/v4/key/$key
Get the content of a key from the repository with the specified uuid. Get the content of a key from the repository with the specified uuid.
@ -144,20 +144,24 @@ an unlocked annexed file, which got modified while it was being sent.
When the key is not present on the server, it will respond When the key is not present on the server, it will respond
with 404 Not Found. with 404 Not Found.
### GET /git-annex/$uuid/v3/key/$key
Identical to v4.
### GET /git-annex/$uuid/v2/key/$key ### GET /git-annex/$uuid/v2/key/$key
Identical to v3. Identical to v4.
### GET /git-annex/$uuid/v1/key/$key ### GET /git-annex/$uuid/v1/key/$key
Identical to v3. Identical to v4.
### GET /git-annex/$uuid/v0/key/$key ### GET /git-annex/$uuid/v0/key/$key
Same as v3, except the X-git-annex-data-length header is not used. Same as v4, except the X-git-annex-data-length header is not used.
Additional checking client-side will be required to validate the data. Additional checking client-side will be required to validate the data.
### POST /git-annex/$uuid/v3/checkpresent ### POST /git-annex/$uuid/v4/checkpresent
Checks if a key is currently present on the server. Checks if a key is currently present on the server.
@ -173,19 +177,23 @@ The body of the request is empty.
The server responds with a JSON object with a "present" field that is true The server responds with a JSON object with a "present" field that is true
if the key is present, or false if it is not present. if the key is present, or false if it is not present.
### POST /git-annex/$uuid/v3/checkpresent
Identical to v4.
### POST /git-annex/$uuid/v2/checkpresent ### POST /git-annex/$uuid/v2/checkpresent
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v1/checkpresent ### POST /git-annex/$uuid/v1/checkpresent
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v0/checkpresent ### POST /git-annex/$uuid/v0/checkpresent
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v3/lockcontent ### POST /git-annex/$uuid/v4/lockcontent
Locks the content of a key on the server, preventing it from being removed. Locks the content of a key on the server, preventing it from being removed.
@ -203,19 +211,23 @@ The key will remain locked for 10 minutes. But, usually `keeplocked`
is used to control the lifetime of the lock, using the "lockid" is used to control the lifetime of the lock, using the "lockid"
parameter from the server's reply. (See below.) parameter from the server's reply. (See below.)
### POST /git-annex/$uuid/v3/lockcontent
Identical to v4.
### POST /git-annex/$uuid/v2/lockcontent ### POST /git-annex/$uuid/v2/lockcontent
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v1/lockcontent ### POST /git-annex/$uuid/v1/lockcontent
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v0/lockcontent ### POST /git-annex/$uuid/v0/lockcontent
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v3/keeplocked ### POST /git-annex/$uuid/v4/keeplocked
Controls the lifetime of a lock on a key that was earlier obtained Controls the lifetime of a lock on a key that was earlier obtained
with `lockcontent`. with `lockcontent`.
@ -252,21 +264,25 @@ locked for 10 minutes from the time it was first locked.
Note that the common parameters bypass and clientuuid, while Note that the common parameters bypass and clientuuid, while
accepted, have no effect. accepted, have no effect.
### POST /git-annex/$uuid/v3/keeplocked
Identical to v4.
### POST /git-annex/$uuid/v2/keeplocked ### POST /git-annex/$uuid/v2/keeplocked
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v1/keeplocked ### POST /git-annex/$uuid/v1/keeplocked
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v0/keeplocked ### POST /git-annex/$uuid/v0/keeplocked
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v3/remove ### POST /git-annex/$uuid/v4/remove
Remove a key's content from the server. Remove a key's content from the repository.
Example: Example:
@ -284,21 +300,25 @@ or false if the key was not able to be removed.
The JSON object can have an additional field "plusuuids" that is a list of The JSON object can have an additional field "plusuuids" that is a list of
UUIDs of other repositories that the content was removed from. UUIDs of other repositories that the content was removed from.
### POST /git-annex/$uuid/v3/remove
Identical to v4.
### POST /git-annex/$uuid/v2/remove ### POST /git-annex/$uuid/v2/remove
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v1/remove ### POST /git-annex/$uuid/v1/remove
Same as v3, except the JSON will not include "plusuuids". Same as v4, except the JSON will not include "plusuuids".
### POST /git-annex/$uuid/v0/remove ### POST /git-annex/$uuid/v0/remove
Identical to v1. Identical to v1.
### POST /git-annex/$uuid/v3/remove-before ### POST /git-annex/$uuid/v4/remove-before
Remove a key's content from the server, but only before a specified time. Remove a key's content from the repository, but only before a specified time.
Example: Example:
@ -314,7 +334,11 @@ removal will fail and the server will respond with: `{"removed": false}`
This is used to avoid removing content after a point in This is used to avoid removing content after a point in
time where it is no longer locked in other repostitories. time where it is no longer locked in other repostitories.
### POST /git-annex/$uuid/v3/gettimestamp ### POST /git-annex/$uuid/v3/remove-before
Identical to v4.
### POST /git-annex/$uuid/v4/gettimestamp
Gets the current timestamp from the server. Gets the current timestamp from the server.
@ -331,9 +355,13 @@ current value of its monotonic clock, as a number of seconds.
Important: If multiple servers are serving this API for the same Important: If multiple servers are serving this API for the same
repository, they MUST all use the same monotonic clock. repository, they MUST all use the same monotonic clock.
### POST /git-annex/$uuid/v3/put ### POST /git-annex/$uuid/v3/gettimestamp
Store content on the server. Identical to v4.
### POST /git-annex/$uuid/v4/put
Store content in the repository.
Example: Example:
@ -358,11 +386,18 @@ There are are also these optional parameters:
Number of bytes that have been omitted from the beginning of the file. Number of bytes that have been omitted from the beginning of the file.
Usually this will be determined by making a `putoffset` request. Usually this will be determined by making a `putoffset` request.
* `data-present`
When set to 1, this indicates that the data has been sent to the repository
in some other way. The body of the request will be empty. The server will
verify that the data is present in the repository and will proceed the
same as if the data was sent in the request.
The `Content-Type` header should be `application/octet-stream`. The `Content-Type` header should be `application/octet-stream`.
The `X-git-annex-data-length` must be included. It indicates the number The `X-git-annex-data-length` must be included. It indicates the number
of bytes of content that are expected to be sent. of bytes of content that are expected to be sent in the body of the
Note that there is no need to send a Content-Length header. request. Note that there is no need to send a Content-Length header.
If the length of the body is different than what the the If the length of the body is different than what the the
X-git-annex-data-length header indicated, then the data is invalid and X-git-annex-data-length header indicated, then the data is invalid and
@ -371,10 +406,16 @@ an unlocked annexed file, which got modified while it was being sent.
The server responds with a JSON object with a field "stored" The server responds with a JSON object with a field "stored"
that is true if it received the data and stored the content. that is true if it received the data and stored the content.
(Or when `data-out-of-band` is used, if it has verified that the key is
present in the repository.)
The JSON object can have an additional field "plusuuids" that is a list of The JSON object can have an additional field "plusuuids" that is a list of
UUIDs of other repositories that the content was stored to. UUIDs of other repositories that the content was stored to.
### POST /git-annex/$uuid/v3/put
Identical to v4 except the `data-out-of-band` parameter cannot be used.
### POST /git-annex/$uuid/v2/put ### POST /git-annex/$uuid/v2/put
Identical to v3. Identical to v3.
@ -387,7 +428,7 @@ Same as v3, except the JSON will not include "plusuuids".
Same as v1, except additional checking is done to validate the data. Same as v1, except additional checking is done to validate the data.
### POST /git-annex/$uuid/v3/putoffset ### POST /git-annex/$uuid/v4/putoffset
Asks the server what `offset` can be used in a `put` of a key. Asks the server what `offset` can be used in a `put` of a key.
@ -419,13 +460,17 @@ part way through a `PUT`, a synthetic empty `DATA` followed by `INVALID`
will be used to get the P2P protocol back into a state where it will accept will be used to get the P2P protocol back into a state where it will accept
any request.] any request.]
### POST /git-annex/$uuid/v3/putoffset
Identical to v4.
### POST /git-annex/$uuid/v2/putoffset ### POST /git-annex/$uuid/v2/putoffset
Identical to v3. Identical to v4.
### POST /git-annex/$uuid/v1/putoffset ### POST /git-annex/$uuid/v1/putoffset
Same as v3, except the JSON will not include "plusuuids". Same as v4, except the JSON will not include "plusuuids".
## parts of P2P protocol that are not supported over HTTP ## parts of P2P protocol that are not supported over HTTP