finalizing HTTP P2P protocol
Managed to avoid netstrings. Actually, using netstrings while streaming lazy ByteString turns out to be very difficult. So instead, have a header that specifies the expected amount of data, and then it can just arrange to send a different amount of data if it needs to indicate INVALID. Also improved the interface for GET of a key.
This commit is contained in:
Joey Hess
parent
5e564947d7
commit
2fb3ef4d41
1 changed files with 86 additions and 102 deletions
|
|
@ -2,11 +2,6 @@
|
|||
|
||||
Draft 1 of a complete [[P2P_protocol]] over HTTP.
|
||||
|
||||
## git-annex protocol endpoint and version
|
||||
|
||||
The git-annex protocol endpoint is "/git-annex" appended to the HTTP
|
||||
url of a git remote.
|
||||
|
||||
## authentication
|
||||
|
||||
A git-annex protocol endpoint can optionally operate in readonly mode without
|
||||
|
|
@ -35,7 +30,8 @@ protocol version.
|
|||
|
||||
## common request parameters
|
||||
|
||||
Every request has some common parameters that are always included:
|
||||
Every request supports these common parameters, and unless documented
|
||||
otherwise, a request requires both of them to be included.
|
||||
|
||||
* `clientuuid`
|
||||
|
||||
|
|
@ -62,28 +58,69 @@ version, to create a P2P session. The P2P session is driven through
|
|||
the AUTH, VERSION, and BYPASS messages, leaving the session ready to
|
||||
service requests.]
|
||||
|
||||
## binary data framing
|
||||
## requests
|
||||
|
||||
When a request body or response body includes binary data, eg the content
|
||||
of a large file, the body is framed using
|
||||
[netstrings](http://cr.yp.to/proto/netstrings.txt).
|
||||
### GET /git-annex/key/$key
|
||||
|
||||
The netstring framing is simply the length of the string in ASCII
|
||||
digits, followed by the string, and then a comma.
|
||||
This is a simple, unversioned interface to get a key from the server.
|
||||
It is not part of the P2P protocol per se, but is provided to let
|
||||
other clients than git-annex easily download the content of keys from the
|
||||
http server.
|
||||
|
||||
This allows efficiently sending binary data in one frame, followed by a
|
||||
second frame that can contain eg a JSON document.
|
||||
This behaves the same as `GET /git-annex/v3/key/$key`, although its
|
||||
behavior may change in later versions.
|
||||
|
||||
For example, a body containing the binary data "foo" followed by
|
||||
a JSON document `{"valid": true}` is framed like this:
|
||||
|
||||
3:foo,15:{"valid": true},
|
||||
### GET /git-annex/v3/key/$key
|
||||
|
||||
## request messages
|
||||
Get the content of a key from the server.
|
||||
|
||||
All the requests below are sent with the HTTP POST method.
|
||||
This is designed so it can be used both by a peer in the P2P protocol,
|
||||
and by a regular HTTP client that just wants to download a file.
|
||||
|
||||
### checkpresent
|
||||
Example:
|
||||
|
||||
> GET /git-annex/v3/key/SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
||||
< X-git-annex-data-length: 3
|
||||
< Content-Type: application/octet-stream
|
||||
<
|
||||
< foo
|
||||
|
||||
The key to get is the part of the url after "/git-annex/vN/key/"
|
||||
and before any url parameters.
|
||||
|
||||
All parameters are optional, including the common parameters, and these:
|
||||
|
||||
* `associatedfile`
|
||||
|
||||
The name of a file in the git repository, for informational purposes
|
||||
only.
|
||||
|
||||
* `offset`
|
||||
|
||||
Number of bytes to skip sending from the beginning of the file.
|
||||
|
||||
Request headers are currently ignored, so eg Range requests are
|
||||
not supported. (This would be possible to implement, up to a point.)
|
||||
|
||||
The body of the request is empty.
|
||||
|
||||
The server's response will have a `Content-Type` header of
|
||||
`application/octet-stream`.
|
||||
|
||||
The server's response will have a `X-git-annex-data-length`
|
||||
header that indicates the number of bytes of content that are expected to
|
||||
be sent. Note that there is no Content-Length header.
|
||||
|
||||
The body of the response is the content of the key.
|
||||
|
||||
If the length of the body is different than what the the
|
||||
X-git-annex-data-length header indicated, then the data is invalid and
|
||||
should not be used. This can happen when eg, the data was being sent from
|
||||
an unlocked annexed file, which got modified while it was being sent.
|
||||
|
||||
When the content is not present, the server will respond with 404.
|
||||
|
||||
### POST /git-annex/v3/checkpresent
|
||||
|
||||
Checks if a key is currently present on the server.
|
||||
|
||||
|
|
@ -99,10 +136,17 @@ The body of the request is empty.
|
|||
The server responds with a JSON object with a "present" field that is true
|
||||
if the key is present, or false if it is not present.
|
||||
|
||||
### lockcontent
|
||||
### POST /git-annex/v3/lockcontent
|
||||
|
||||
Locks the content of a key on the server, preventing it from being removed.
|
||||
|
||||
Example:
|
||||
|
||||
> POST /git-annex/v3/lockcontent?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
||||
[websocket protocol follows]
|
||||
< SUCCESS
|
||||
> UNLOCKCONTENT
|
||||
|
||||
There is one required additional parameter, `key`.
|
||||
|
||||
This request opens a websocket between the client and the server.
|
||||
|
|
@ -116,7 +160,7 @@ If the client disconnects without sending "UNLOCKCONTENT", or the web
|
|||
server gets shut down before it can receive that, the content will remain
|
||||
locked for at least 10 minutes from when the server sent "SUCCESS".
|
||||
|
||||
### remove
|
||||
### POST /git-annex/v3/remove
|
||||
|
||||
Remove a key's content from the server.
|
||||
|
||||
|
|
@ -140,7 +184,7 @@ If the server does not allow removing the key due to a policy
|
|||
(eg due to being read-only or append-only), it will respond with a JSON
|
||||
object with an "error" field that has an error message as its value.
|
||||
|
||||
## remove-before
|
||||
## POST /git-annex/v3/remove-before
|
||||
|
||||
Remove a key's content from the server, but only before a specified time.
|
||||
|
||||
|
|
@ -158,7 +202,7 @@ removal will fail and the server will respond with: `{"removed": false}`
|
|||
This is used to avoid removing content after a point in
|
||||
time where it is no longer locked in other repostitories.
|
||||
|
||||
## gettimestamp
|
||||
## POST /git-annex/v3/gettimestamp
|
||||
|
||||
Gets the current timestamp from the server.
|
||||
|
||||
|
|
@ -175,7 +219,7 @@ current value of its monotonic clock, as a number of seconds.
|
|||
Important: If multiple servers are serving this protocol for the same
|
||||
repository, they MUST all use the same monotonic clock.
|
||||
|
||||
### put
|
||||
### POST /git-annex/v3/put
|
||||
|
||||
Store content on the server.
|
||||
|
||||
|
|
@ -183,8 +227,9 @@ Example:
|
|||
|
||||
> POST /git-annex/v3/put?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
||||
> Content-Type: application/octet-stream
|
||||
> Content-Length: 25
|
||||
> 3:foo,15:{"valid": true},
|
||||
> X-git-annex-object-size: 3
|
||||
>
|
||||
> foo
|
||||
< {"stored": true}
|
||||
|
||||
There is one required additional parameter, `key`.
|
||||
|
|
@ -201,21 +246,16 @@ There are are also these optional parameters:
|
|||
Number of bytes that have been omitted from the beginning of the file.
|
||||
Usually this will be determined by making a `putoffset` request.
|
||||
|
||||
The body of the request is two items framed with netstrings.
|
||||
|
||||
The first item is the content of the key, starting from the specified
|
||||
offset or from the beginning when no offset was specified.
|
||||
|
||||
The second item is a JSON object.
|
||||
|
||||
The JSON object has a field "valid" that is true when the content was not
|
||||
changed while it was being sent, or false when modified content was sent
|
||||
and should be disregarded by the server. (This corresponds to the `VALID`
|
||||
and `INVALID` messages in the P2P protocol.)
|
||||
|
||||
The `Content-Type` header should be `application/octet-stream`.
|
||||
|
||||
The `Content-Length` header should be set to the length of the body.
|
||||
The `X-git-annex-data-length` must be included. It indicates the number
|
||||
of bytes of content that are expected to be sent.
|
||||
Note that there is no need to send a Content-Length header.
|
||||
|
||||
If the length of the body is different than what the the
|
||||
X-git-annex-data-length header indicated, then the data is invalid and
|
||||
should not be used. This can happen when eg, the data was being sent from
|
||||
an unlocked annexed file, which got modified while it was being sent.
|
||||
|
||||
The server responds with a JSON object with a field "stored"
|
||||
that is true if it received the data and stored the
|
||||
|
|
@ -224,11 +264,12 @@ content.
|
|||
The JSON object can have an additional field "plusuuids" that is a list of
|
||||
UUIDs of other repositories that the content was stored to.
|
||||
|
||||
If the server does not allow storing the key due to a policy
|
||||
(eg due to being read-only or append-only), it will respond with a JSON
|
||||
object with an "error" field that has an error message as its value.
|
||||
If the server does not allow storing the key due eg to a policy
|
||||
(eg due to being read-only or append-only), or due to the data being
|
||||
invalid, or because it ran out of disk space, it will respond with a
|
||||
JSON object with an "error" field that has an error message as its value.
|
||||
|
||||
### putoffset
|
||||
### POST /git-annex/v3/putoffset
|
||||
|
||||
Asks the server what `offset` can be used in a `put` of a key.
|
||||
|
||||
|
|
@ -258,63 +299,6 @@ part way through a `PUT`, a synthetic empty `DATA` followed by `INVALID`
|
|||
will be used to get the P2P protocol back into a state where it will accept
|
||||
any request.]
|
||||
|
||||
### get
|
||||
|
||||
Get content from the server.
|
||||
|
||||
Example:
|
||||
|
||||
> POST /git-annex/v3/get?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
||||
< Content-Type: application/octet-stream
|
||||
> Content-Length: 20
|
||||
> 3:foo,15:{"valid": true},
|
||||
|
||||
There is one required additional parameter, `key`.
|
||||
|
||||
There is are also these optional parameters:
|
||||
|
||||
* `associatedfile`
|
||||
|
||||
The name of a file in the git repository, for informational purposes
|
||||
only.
|
||||
|
||||
* `offset`
|
||||
|
||||
Number of bytes to skip sending from the beginning of the file.
|
||||
|
||||
The body of the request is empty.
|
||||
|
||||
The server's response will have a `Content-Type` header of
|
||||
`application/octet-stream`.
|
||||
|
||||
The server's response will have a `Content-Length` header
|
||||
set to the length of the body.
|
||||
|
||||
The body of the response is two items framed with netstrings.
|
||||
|
||||
The first item is the content of the key, starting from the specified
|
||||
offset or from the beginning when no offset was specified.
|
||||
|
||||
The second item is a JSON object.
|
||||
|
||||
The JSON object has a field "valid" that is true when the content
|
||||
was not changed while it was being sent, or false when whatever
|
||||
content was sent is not the actual content of the key and should be
|
||||
disregared. (This corresponds to the `VALID` and `INVALID` messages
|
||||
in the P2P protocol.)
|
||||
|
||||
## simple HTTP GET
|
||||
|
||||
The git-annex protocol endpoint also supports a regular HTTP get
|
||||
of a key. This is not part of the P2P protocol, but is provided for
|
||||
convenience, to allow other clients than git-annex to easily download
|
||||
the content of a key.
|
||||
|
||||
> GET /git-annex/key/SHA1--foo HTTP/1.1
|
||||
< Content-Type: application/octet-stream
|
||||
< Content-Length: 3
|
||||
< foo
|
||||
|
||||
## parts of P2P protocol that are not supported over HTTP
|
||||
|
||||
`NOTIFYCHANGE` is not supported, but it would be possible to extend
|
||||
|
|
|
|||
Loading…
Reference in a new issue