40306d3fcf
Added v2-v0 endpoints. These are tedious, but will be needed in order to use the HTTP protocol to proxy to repositories with older git-annex, where git-annex-shell will be speaking an older version of the protocol. Changed GET to use 422 when the content is not present. 404 is needed to detect when a protocol version is not supported.
389 lines
12 KiB
Markdown
389 lines
12 KiB
Markdown
[[!toc ]]
|
|
|
|
Draft 1 of a complete [[P2P_protocol]] over HTTP.
|
|
|
|
## authentication
|
|
|
|
A git-annex protocol endpoint can optionally operate in readonly mode without
|
|
authentication.
|
|
|
|
Authentication is required to make any changes.
|
|
|
|
Authentication is done using HTTP basic auth.
|
|
|
|
The user is recommended to only authenticate over HTTPS, since otherwise
|
|
HTTP basic auth (as well as git-annex data) can be snooped. But some users
|
|
may want git-annex to use HTTP in eg a LAN.
|
|
|
|
## protocol version
|
|
|
|
Each request in the protocol is versioned. The versions correspond
|
|
to P2P protocol versions.
|
|
|
|
The protocol version comes before the request. Eg: `/git-annex/v3/put`
|
|
|
|
If the server does not support a particular protocol version, the
|
|
request will fail with a 404, and the client should fall back to an earlier
|
|
protocol version.
|
|
|
|
## common request parameters
|
|
|
|
Every request supports these common parameters, and unless documented
|
|
otherwise, a request requires both of them to be included.
|
|
|
|
* `clientuuid`
|
|
|
|
The value is the UUID of the git-annex repository of the client.
|
|
|
|
* `serveruuid`
|
|
|
|
The value is the UUID of the git-annex repository that the server
|
|
should serve.
|
|
|
|
Any request may also optionally include these parameters:
|
|
|
|
* `bypass`
|
|
|
|
The value is the UUID of a cluster gateway, which the server should avoid
|
|
connecting to when serving a cluster. This is the equivilant of the
|
|
`BYPASS` message in the [[P2P_Protocol]].
|
|
|
|
This parameter can be given multiple times to list several cluster
|
|
gateway UUIDs.
|
|
|
|
This parameter is only available for v3 and above.
|
|
|
|
[Internally, git-annex can use these common parameters, plus the protocol
|
|
version, to create a P2P session. The P2P session is driven through
|
|
the AUTH, VERSION, and BYPASS messages, leaving the session ready to
|
|
service requests.]
|
|
|
|
## requests
|
|
|
|
### GET /git-annex/key/$key
|
|
|
|
This is a simple, unversioned interface to get a key from the server.
|
|
It is not part of the P2P protocol per se, but is provided to let
|
|
other clients than git-annex easily download the content of keys from the
|
|
http server.
|
|
|
|
This behaves almost the same as `GET /git-annex/v3/key/$key`, although its
|
|
behavior may change in later versions.
|
|
|
|
When the key is not present on the server, this returns a 404 Not Found.
|
|
|
|
### GET /git-annex/v3/key/$key
|
|
|
|
Get the content of a key from the server.
|
|
|
|
This is designed so it can be used both by a peer in the P2P protocol,
|
|
and by a regular HTTP client that just wants to download a file.
|
|
|
|
Example:
|
|
|
|
> GET /git-annex/v3/key/SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
|
< X-git-annex-data-length: 3
|
|
< Content-Type: application/octet-stream
|
|
<
|
|
< foo
|
|
|
|
The key to get is the part of the url after "/git-annex/vN/key/"
|
|
and before any url parameters.
|
|
|
|
All parameters are optional, including the common parameters, and these:
|
|
|
|
* `associatedfile`
|
|
|
|
The name of a file in the git repository, for informational purposes
|
|
only.
|
|
|
|
* `offset`
|
|
|
|
Number of bytes to skip sending from the beginning of the file.
|
|
|
|
Request headers are currently ignored, so eg Range requests are
|
|
not supported. (This would be possible to implement, up to a point.)
|
|
|
|
The body of the request is empty.
|
|
|
|
The server's response will have a `Content-Type` header of
|
|
`application/octet-stream`.
|
|
|
|
The server's response will have a `X-git-annex-data-length`
|
|
header that indicates the number of bytes of content that are expected to
|
|
be sent. Note that there is no Content-Length header.
|
|
|
|
The body of the response is the content of the key.
|
|
|
|
If the length of the body is different than what the the
|
|
X-git-annex-data-length header indicated, then the data is invalid and
|
|
should not be used. This can happen when eg, the data was being sent from
|
|
an unlocked annexed file, which got modified while it was being sent.
|
|
|
|
When the content is not present, the server will respond with
|
|
422 Unprocessable Content.
|
|
|
|
### GET /git-annex/v2/key/$key
|
|
|
|
Identical to v3.
|
|
|
|
### GET /git-annex/v1/key/$key
|
|
|
|
Identical to v3.
|
|
|
|
### GET /git-annex/v0/key/$key
|
|
|
|
Same as v3, except there is no X-git-annex-data-length header.
|
|
Additional checking client-side will be required to validate the data.
|
|
|
|
### POST /git-annex/v3/checkpresent
|
|
|
|
Checks if a key is currently present on the server.
|
|
|
|
Example:
|
|
|
|
> POST /git-annex/v3/checkpresent?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
|
< {"present": true}
|
|
|
|
There is one required additional parameter, `key`.
|
|
|
|
The body of the request is empty.
|
|
|
|
The server responds with a JSON object with a "present" field that is true
|
|
if the key is present, or false if it is not present.
|
|
|
|
### POST /git-annex/v2/checkpresent
|
|
|
|
Identical to v3.
|
|
|
|
### POST /git-annex/v1/checkpresent
|
|
|
|
Identical to v3.
|
|
|
|
### POST /git-annex/v0/checkpresent
|
|
|
|
Identical to v3.
|
|
|
|
### POST /git-annex/v3/lockcontent
|
|
|
|
Locks the content of a key on the server, preventing it from being removed.
|
|
|
|
Example:
|
|
|
|
> POST /git-annex/v3/lockcontent?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
|
[websocket protocol follows]
|
|
< SUCCESS
|
|
> UNLOCKCONTENT
|
|
|
|
There is one required additional parameter, `key`.
|
|
|
|
This request opens a websocket between the client and the server.
|
|
The server sends "SUCCESS" over the websocket once it has locked
|
|
the content. Or it sends "FAILURE" if it is unable to lock the content.
|
|
|
|
Once the server has sent "SUCCESS", the content remains locked
|
|
until the client sends "UNLOCKCONTENT" over the websocket.
|
|
|
|
If the client disconnects without sending "UNLOCKCONTENT", or the web
|
|
server gets shut down before it can receive that, the content will remain
|
|
locked for at least 10 minutes from when the server sent "SUCCESS".
|
|
|
|
### POST /git-annex/v2/lockcontent
|
|
|
|
Identical to v3.
|
|
|
|
### POST /git-annex/v1/lockcontent
|
|
|
|
Identical to v3.
|
|
|
|
### POST /git-annex/v0/lockcontent
|
|
|
|
Identical to v3.
|
|
|
|
### POST /git-annex/v3/remove
|
|
|
|
Remove a key's content from the server.
|
|
|
|
Example:
|
|
|
|
> POST /git-annex/v3/remove?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
|
< {"removed": true}
|
|
|
|
There is one required additional parameter, `key`.
|
|
|
|
The body of the request is empty.
|
|
|
|
The server responds with a JSON object with a "removed" field that is true
|
|
if the key was removed (or was not present on the server),
|
|
or false if the key was not able to be removed.
|
|
|
|
The JSON object can have an additional field "plusuuids" that is a list of
|
|
UUIDs of other repositories that the content was removed from.
|
|
|
|
If the server does not allow removing the key due to a policy
|
|
(eg due to being read-only or append-only), it will respond with a JSON
|
|
object with an "error" field that has an error message as its value.
|
|
|
|
### POST /git-annex/v2/remove
|
|
|
|
Identical to v3.
|
|
|
|
### POST /git-annex/v1/remove
|
|
|
|
Same as v3, except the JSON will not include "plusuuids".
|
|
|
|
### POST /git-annex/v0/remove
|
|
|
|
Identival to v1.
|
|
|
|
## POST /git-annex/v3/remove-before
|
|
|
|
Remove a key's content from the server, but only before a specified time.
|
|
|
|
Example:
|
|
|
|
> POST /git-annex/v3/remove-before?timestamp=4949292929&key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
|
< {"removed": true}
|
|
|
|
This is the same as the `remove` request, but with an additional parameter,
|
|
`timestamp`.
|
|
|
|
If the server's monotonic clock is past the specified timestamp, the
|
|
removal will fail and the server will respond with: `{"removed": false}`
|
|
|
|
This is used to avoid removing content after a point in
|
|
time where it is no longer locked in other repostitories.
|
|
|
|
## POST /git-annex/v3/gettimestamp
|
|
|
|
Gets the current timestamp from the server.
|
|
|
|
Example:
|
|
|
|
> POST /git-annex/v3/gettimestamp?clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
|
< {"timestamp": 59459392}
|
|
|
|
The body of the request is empty.
|
|
|
|
The server responds with JSON object with a timestmap field that has the
|
|
current value of its monotonic clock, as a number of seconds.
|
|
|
|
Important: If multiple servers are serving this protocol for the same
|
|
repository, they MUST all use the same monotonic clock.
|
|
|
|
### POST /git-annex/v3/put
|
|
|
|
Store content on the server.
|
|
|
|
Example:
|
|
|
|
> POST /git-annex/v3/put?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
|
> Content-Type: application/octet-stream
|
|
> X-git-annex-object-size: 3
|
|
>
|
|
> foo
|
|
< {"stored": true}
|
|
|
|
There is one required additional parameter, `key`.
|
|
|
|
There are are also these optional parameters:
|
|
|
|
* `associatedfile`
|
|
|
|
The name of a file in the git repository, for informational purposes
|
|
only.
|
|
|
|
* `offset`
|
|
|
|
Number of bytes that have been omitted from the beginning of the file.
|
|
Usually this will be determined by making a `putoffset` request.
|
|
|
|
The `Content-Type` header should be `application/octet-stream`.
|
|
|
|
The `X-git-annex-data-length` must be included. It indicates the number
|
|
of bytes of content that are expected to be sent.
|
|
Note that there is no need to send a Content-Length header.
|
|
|
|
If the length of the body is different than what the the
|
|
X-git-annex-data-length header indicated, then the data is invalid and
|
|
should not be used. This can happen when eg, the data was being sent from
|
|
an unlocked annexed file, which got modified while it was being sent.
|
|
|
|
The server responds with a JSON object with a field "stored"
|
|
that is true if it received the data and stored the
|
|
content.
|
|
|
|
The JSON object can have an additional field "plusuuids" that is a list of
|
|
UUIDs of other repositories that the content was stored to.
|
|
|
|
If the server does not allow storing the key due eg to a policy
|
|
(eg due to being read-only or append-only), or due to the data being
|
|
invalid, or because it ran out of disk space, it will respond with a
|
|
JSON object with an "error" field that has an error message as its value.
|
|
|
|
### POST /git-annex/v2/put
|
|
|
|
Identical to v3.
|
|
|
|
### POST /git-annex/v1/put
|
|
|
|
Same as v3, except the JSON will not include "plusuuids".
|
|
|
|
### POST /git-annex/v0/put
|
|
|
|
Same as v1, except there is no X-git-annex-data-length header.
|
|
Additional checking client-side will be required to validate the data.
|
|
|
|
### POST /git-annex/v3/putoffset
|
|
|
|
Asks the server what `offset` can be used in a `put` of a key.
|
|
|
|
This should usually be used right before sending a `put` request.
|
|
The offset may not be valid after some point in time, which could result in
|
|
the `put` request failing.
|
|
|
|
Example:
|
|
|
|
> POST /git-annex/v3/putoffset?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
|
< {"offset": 10}
|
|
|
|
There is one required additional parameter, `key`.
|
|
|
|
The body of the request is empty.
|
|
|
|
The server responds with a JSON object with an "offset" field that
|
|
is the largest allowable offset.
|
|
|
|
If the server already has the content of the key, it will respond with a
|
|
JSON object with an "alreadyhave" field that is set to true. This JSON
|
|
object may also have a field "plusuuids" that lists
|
|
the UUIDs of other repositories where the content is stored, in addition to
|
|
the serveruuid.
|
|
|
|
If the server does not allow storing the key due to a policy
|
|
(eg due to being read-only or append-only), it will respond with a JSON
|
|
object with an "error" field that has an error message as its value.
|
|
|
|
[Implementation note: This will be implemented by sending `PUT` and
|
|
returning the `PUT-FROM` offset. To avoid leaving the P2P protocol stuck
|
|
part way through a `PUT`, a synthetic empty `DATA` followed by `INVALID`
|
|
will be used to get the P2P protocol back into a state where it will accept
|
|
any request.]
|
|
|
|
### POST /git-annex/v2/putoffset
|
|
|
|
Identical to v3.
|
|
|
|
### POST /git-annex/v1/putoffset
|
|
|
|
Same as v3, except the JSON will not include "plusuuids".
|
|
|
|
## parts of P2P protocol that are not supported over HTTP
|
|
|
|
`NOTIFYCHANGE` is not supported, but it would be possible to extend
|
|
this HTTP protocol to support it.
|
|
|
|
`CONNECT` is not supported, and due to the bi-directional message passing
|
|
nature of it, it cannot easily be done over HTTP (would need websockets).
|
|
It should not be necessary anyway, because the git repository itself can be
|
|
accessed over HTTP.
|