mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

finalizing HTTP P2P protocol

Browse source 
Managed to avoid netstrings. Actually, using netstrings while streaming
lazy ByteString turns out to be very difficult. So instead, have a
header that specifies the expected amount of data, and then it can just
arrange to send a different amount of data if it needs to indicate
INVALID.

Also improved the interface for GET of a key.
This commit is contained in:
Joey Hess 2024-07-05 15:00:05 -04:00
parent 5e564947d7
commit 2fb3ef4d41
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
1 changed files with 86 additions and 102 deletions
Download patch file Download diff file Expand all files Collapse all files

188
doc/design/p2p_protocol_over_http/draft1.mdwn
View file

@ -2,11 +2,6 @@
Draft 1 of a complete [[P2P_protocol]] over HTTP. Draft 1 of a complete [[P2P_protocol]] over HTTP.
## git-annex protocol endpoint and version
The git-annex protocol endpoint is "/git-annex" appended to the HTTP
url of a git remote.
## authentication ## authentication
A git-annex protocol endpoint can optionally operate in readonly mode without A git-annex protocol endpoint can optionally operate in readonly mode without
@ -35,7 +30,8 @@ protocol version.
## common request parameters ## common request parameters
Every request has some common parameters that are always included: Every request supports these common parameters, and unless documented
otherwise, a request requires both of them to be included.
* `clientuuid` * `clientuuid`
@ -62,28 +58,69 @@ version, to create a P2P session. The P2P session is driven through
the AUTH, VERSION, and BYPASS messages, leaving the session ready to the AUTH, VERSION, and BYPASS messages, leaving the session ready to
service requests.] service requests.]
## binary data framing ## requests
When a request body or response body includes binary data, eg the content ### GET /git-annex/key/$key
of a large file, the body is framed using
[netstrings](http://cr.yp.to/proto/netstrings.txt).
The netstring framing is simply the length of the string in ASCII This is a simple, unversioned interface to get a key from the server.
digits, followed by the string, and then a comma. It is not part of the P2P protocol per se, but is provided to let
other clients than git-annex easily download the content of keys from the
http server.
This allows efficiently sending binary data in one frame, followed by a This behaves the same as `GET /git-annex/v3/key/$key`, although its
second frame that can contain eg a JSON document. behavior may change in later versions.
For example, a body containing the binary data "foo" followed by ### GET /git-annex/v3/key/$key
a JSON document `{"valid": true}` is framed like this:
3:foo,15:{"valid": true},
## request messages Get the content of a key from the server.
All the requests below are sent with the HTTP POST method. This is designed so it can be used both by a peer in the P2P protocol,
and by a regular HTTP client that just wants to download a file.
### checkpresent Example:
> GET /git-annex/v3/key/SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
< X-git-annex-data-length: 3
< Content-Type: application/octet-stream
<
< foo
The key to get is the part of the url after "/git-annex/vN/key/"
and before any url parameters.
All parameters are optional, including the common parameters, and these:
* `associatedfile`
The name of a file in the git repository, for informational purposes
only.
* `offset`
Number of bytes to skip sending from the beginning of the file.
Request headers are currently ignored, so eg Range requests are
not supported. (This would be possible to implement, up to a point.)
The body of the request is empty.
The server's response will have a `Content-Type` header of
`application/octet-stream`.
The server's response will have a `X-git-annex-data-length`
header that indicates the number of bytes of content that are expected to
be sent. Note that there is no Content-Length header.
The body of the response is the content of the key.
If the length of the body is different than what the the
X-git-annex-data-length header indicated, then the data is invalid and
should not be used. This can happen when eg, the data was being sent from
an unlocked annexed file, which got modified while it was being sent.
When the content is not present, the server will respond with 404.
### POST /git-annex/v3/checkpresent
Checks if a key is currently present on the server. Checks if a key is currently present on the server.
@ -99,10 +136,17 @@ The body of the request is empty.
The server responds with a JSON object with a "present" field that is true The server responds with a JSON object with a "present" field that is true
if the key is present, or false if it is not present. if the key is present, or false if it is not present.
### lockcontent ### POST /git-annex/v3/lockcontent
Locks the content of a key on the server, preventing it from being removed. Locks the content of a key on the server, preventing it from being removed.
Example:
> POST /git-annex/v3/lockcontent?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
[websocket protocol follows]
< SUCCESS
> UNLOCKCONTENT
There is one required additional parameter, `key`. There is one required additional parameter, `key`.
This request opens a websocket between the client and the server. This request opens a websocket between the client and the server.
@ -116,7 +160,7 @@ If the client disconnects without sending "UNLOCKCONTENT", or the web
server gets shut down before it can receive that, the content will remain server gets shut down before it can receive that, the content will remain
locked for at least 10 minutes from when the server sent "SUCCESS". locked for at least 10 minutes from when the server sent "SUCCESS".
### remove ### POST /git-annex/v3/remove
Remove a key's content from the server. Remove a key's content from the server.
@ -140,7 +184,7 @@ If the server does not allow removing the key due to a policy
(eg due to being read-only or append-only), it will respond with a JSON (eg due to being read-only or append-only), it will respond with a JSON
object with an "error" field that has an error message as its value. object with an "error" field that has an error message as its value.
## remove-before ## POST /git-annex/v3/remove-before
Remove a key's content from the server, but only before a specified time. Remove a key's content from the server, but only before a specified time.
@ -158,7 +202,7 @@ removal will fail and the server will respond with: `{"removed": false}`
This is used to avoid removing content after a point in This is used to avoid removing content after a point in
time where it is no longer locked in other repostitories. time where it is no longer locked in other repostitories.
## gettimestamp ## POST /git-annex/v3/gettimestamp
Gets the current timestamp from the server. Gets the current timestamp from the server.
@ -175,7 +219,7 @@ current value of its monotonic clock, as a number of seconds.
Important: If multiple servers are serving this protocol for the same Important: If multiple servers are serving this protocol for the same
repository, they MUST all use the same monotonic clock. repository, they MUST all use the same monotonic clock.
### put ### POST /git-annex/v3/put
Store content on the server. Store content on the server.
@ -183,8 +227,9 @@ Example:
> POST /git-annex/v3/put?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1 > POST /git-annex/v3/put?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
> Content-Type: application/octet-stream > Content-Type: application/octet-stream
> Content-Length: 25 > X-git-annex-object-size: 3
> 3:foo,15:{"valid": true}, >
> foo
< {"stored": true} < {"stored": true}
There is one required additional parameter, `key`. There is one required additional parameter, `key`.
@ -201,21 +246,16 @@ There are are also these optional parameters:
Number of bytes that have been omitted from the beginning of the file. Number of bytes that have been omitted from the beginning of the file.
Usually this will be determined by making a `putoffset` request. Usually this will be determined by making a `putoffset` request.
The body of the request is two items framed with netstrings.
The first item is the content of the key, starting from the specified
offset or from the beginning when no offset was specified.
The second item is a JSON object.
The JSON object has a field "valid" that is true when the content was not
changed while it was being sent, or false when modified content was sent
and should be disregarded by the server. (This corresponds to the `VALID`
and `INVALID` messages in the P2P protocol.)
The `Content-Type` header should be `application/octet-stream`. The `Content-Type` header should be `application/octet-stream`.
The `Content-Length` header should be set to the length of the body. The `X-git-annex-data-length` must be included. It indicates the number
of bytes of content that are expected to be sent.
Note that there is no need to send a Content-Length header.
If the length of the body is different than what the the
X-git-annex-data-length header indicated, then the data is invalid and
should not be used. This can happen when eg, the data was being sent from
an unlocked annexed file, which got modified while it was being sent.
The server responds with a JSON object with a field "stored" The server responds with a JSON object with a field "stored"
that is true if it received the data and stored the that is true if it received the data and stored the
@ -224,11 +264,12 @@ content.
The JSON object can have an additional field "plusuuids" that is a list of The JSON object can have an additional field "plusuuids" that is a list of
UUIDs of other repositories that the content was stored to. UUIDs of other repositories that the content was stored to.
If the server does not allow storing the key due to a policy If the server does not allow storing the key due eg to a policy
(eg due to being read-only or append-only), it will respond with a JSON (eg due to being read-only or append-only), or due to the data being
object with an "error" field that has an error message as its value. invalid, or because it ran out of disk space, it will respond with a
JSON object with an "error" field that has an error message as its value.
### putoffset ### POST /git-annex/v3/putoffset
Asks the server what `offset` can be used in a `put` of a key. Asks the server what `offset` can be used in a `put` of a key.
@ -258,63 +299,6 @@ part way through a `PUT`, a synthetic empty `DATA` followed by `INVALID`
will be used to get the P2P protocol back into a state where it will accept will be used to get the P2P protocol back into a state where it will accept
any request.] any request.]
### get
Get content from the server.
Example:
> POST /git-annex/v3/get?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
< Content-Type: application/octet-stream
> Content-Length: 20
> 3:foo,15:{"valid": true},
There is one required additional parameter, `key`.
There is are also these optional parameters:
* `associatedfile`
The name of a file in the git repository, for informational purposes
only.
* `offset`
Number of bytes to skip sending from the beginning of the file.
The body of the request is empty.
The server's response will have a `Content-Type` header of
`application/octet-stream`.
The server's response will have a `Content-Length` header
set to the length of the body.
The body of the response is two items framed with netstrings.
The first item is the content of the key, starting from the specified
offset or from the beginning when no offset was specified.
The second item is a JSON object.
The JSON object has a field "valid" that is true when the content
was not changed while it was being sent, or false when whatever
content was sent is not the actual content of the key and should be
disregared. (This corresponds to the `VALID` and `INVALID` messages
in the P2P protocol.)
## simple HTTP GET
The git-annex protocol endpoint also supports a regular HTTP get
of a key. This is not part of the P2P protocol, but is provided for
convenience, to allow other clients than git-annex to easily download
the content of a key.
> GET /git-annex/key/SHA1--foo HTTP/1.1
< Content-Type: application/octet-stream
< Content-Length: 3
< foo
## parts of P2P protocol that are not supported over HTTP ## parts of P2P protocol that are not supported over HTTP
`NOTIFYCHANGE` is not supported, but it would be possible to extend `NOTIFYCHANGE` is not supported, but it would be possible to extend