[[!toc ]] Draft 1 of a complete [[P2P_protocol]] over HTTP. ## authentication A git-annex protocol endpoint can optionally operate in readonly mode without authentication. Authentication is required to make any changes. Authentication is done using HTTP basic auth. The user is recommended to only authenticate over HTTPS, since otherwise HTTP basic auth (as well as git-annex data) can be snooped. But some users may want git-annex to use HTTP in eg a LAN. ## protocol version Each request in the protocol is versioned. The versions correspond to P2P protocol versions, but for simplicity, the minimum version supported over HTTP is version 3. Every implementation of the HTTP protocol must support version 3. The protocol version comes before the request. Eg: `/git-annex/v3/put` If the server does not support a particular protocol version, the request will fail with a 404, and the client should fall back to an earlier protocol version. ## common request parameters Every request supports these common parameters, and unless documented otherwise, a request requires both of them to be included. * `clientuuid` The value is the UUID of the git-annex repository of the client. * `serveruuid` The value is the UUID of the git-annex repository that the server should serve. Any request may also optionally include these parameters: * `bypass` The value is the UUID of a cluster gateway, which the server should avoid connecting to when serving a cluster. This is the equivilant of the `BYPASS` message in the [[P2P_Protocol]]. This parameter can be given multiple times to list several cluster gateway UUIDs. [Internally, git-annex can use these common parameters, plus the protocol version, to create a P2P session. The P2P session is driven through the AUTH, VERSION, and BYPASS messages, leaving the session ready to service requests.] ## requests ### GET /git-annex/key/$key This is a simple, unversioned interface to get a key from the server. It is not part of the P2P protocol per se, but is provided to let other clients than git-annex easily download the content of keys from the http server. This behaves the same as `GET /git-annex/v3/key/$key`, although its behavior may change in later versions. ### GET /git-annex/v3/key/$key Get the content of a key from the server. This is designed so it can be used both by a peer in the P2P protocol, and by a regular HTTP client that just wants to download a file. Example: > GET /git-annex/v3/key/SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1 < X-git-annex-data-length: 3 < Content-Type: application/octet-stream < < foo The key to get is the part of the url after "/git-annex/vN/key/" and before any url parameters. All parameters are optional, including the common parameters, and these: * `associatedfile` The name of a file in the git repository, for informational purposes only. * `offset` Number of bytes to skip sending from the beginning of the file. Request headers are currently ignored, so eg Range requests are not supported. (This would be possible to implement, up to a point.) The body of the request is empty. The server's response will have a `Content-Type` header of `application/octet-stream`. The server's response will have a `X-git-annex-data-length` header that indicates the number of bytes of content that are expected to be sent. Note that there is no Content-Length header. The body of the response is the content of the key. If the length of the body is different than what the the X-git-annex-data-length header indicated, then the data is invalid and should not be used. This can happen when eg, the data was being sent from an unlocked annexed file, which got modified while it was being sent. When the content is not present, the server will respond with 404. ### POST /git-annex/v3/checkpresent Checks if a key is currently present on the server. Example: > POST /git-annex/v3/checkpresent?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1 < {"present": true} There is one required additional parameter, `key`. The body of the request is empty. The server responds with a JSON object with a "present" field that is true if the key is present, or false if it is not present. ### POST /git-annex/v3/lockcontent Locks the content of a key on the server, preventing it from being removed. Example: > POST /git-annex/v3/lockcontent?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1 [websocket protocol follows] < SUCCESS > UNLOCKCONTENT There is one required additional parameter, `key`. This request opens a websocket between the client and the server. The server sends "SUCCESS" over the websocket once it has locked the content. Or it sends "FAILURE" if it is unable to lock the content. Once the server has sent "SUCCESS", the content remains locked until the client sends "UNLOCKCONTENT" over the websocket. If the client disconnects without sending "UNLOCKCONTENT", or the web server gets shut down before it can receive that, the content will remain locked for at least 10 minutes from when the server sent "SUCCESS". ### POST /git-annex/v3/remove Remove a key's content from the server. Example: > POST /git-annex/v3/remove?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1 < {"removed": true} There is one required additional parameter, `key`. The body of the request is empty. The server responds with a JSON object with a "removed" field that is true if the key was removed (or was not present on the server), or false if the key was not able to be removed. The JSON object can have an additional field "plusuuids" that is a list of UUIDs of other repositories that the content was removed from. If the server does not allow removing the key due to a policy (eg due to being read-only or append-only), it will respond with a JSON object with an "error" field that has an error message as its value. ## POST /git-annex/v3/remove-before Remove a key's content from the server, but only before a specified time. Example: > POST /git-annex/v3/remove-before?timestamp=4949292929&key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1 < {"removed": true} This is the same as the `remove` request, but with an additional parameter, `timestamp`. If the server's monotonic clock is past the specified timestamp, the removal will fail and the server will respond with: `{"removed": false}` This is used to avoid removing content after a point in time where it is no longer locked in other repostitories. ## POST /git-annex/v3/gettimestamp Gets the current timestamp from the server. Example: > POST /git-annex/v3/gettimestamp?clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1 < {"timestamp": 59459392} The body of the request is empty. The server responds with JSON object with a timestmap field that has the current value of its monotonic clock, as a number of seconds. Important: If multiple servers are serving this protocol for the same repository, they MUST all use the same monotonic clock. ### POST /git-annex/v3/put Store content on the server. Example: > POST /git-annex/v3/put?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1 > Content-Type: application/octet-stream > X-git-annex-object-size: 3 > > foo < {"stored": true} There is one required additional parameter, `key`. There are are also these optional parameters: * `associatedfile` The name of a file in the git repository, for informational purposes only. * `offset` Number of bytes that have been omitted from the beginning of the file. Usually this will be determined by making a `putoffset` request. The `Content-Type` header should be `application/octet-stream`. The `X-git-annex-data-length` must be included. It indicates the number of bytes of content that are expected to be sent. Note that there is no need to send a Content-Length header. If the length of the body is different than what the the X-git-annex-data-length header indicated, then the data is invalid and should not be used. This can happen when eg, the data was being sent from an unlocked annexed file, which got modified while it was being sent. The server responds with a JSON object with a field "stored" that is true if it received the data and stored the content. The JSON object can have an additional field "plusuuids" that is a list of UUIDs of other repositories that the content was stored to. If the server does not allow storing the key due eg to a policy (eg due to being read-only or append-only), or due to the data being invalid, or because it ran out of disk space, it will respond with a JSON object with an "error" field that has an error message as its value. ### POST /git-annex/v3/putoffset Asks the server what `offset` can be used in a `put` of a key. This should usually be used right before sending a `put` request. The offset may not be valid after some point in time, which could result in the `put` request failing. Example: > POST /git-annex/v3/putoffset?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1 < {"offset": 10} There is one required additional parameter, `key`. The body of the request is empty. The server responds with a JSON object with an "offset" field that is the largest allowable offset. If the server does not allow storing the key due to a policy (eg due to being read-only or append-only), it will respond with a JSON object with an "error" field that has an error message as its value. [Implementation note: This will be implemented by sending `PUT` and returning the `PUT-FROM` offset. To avoid leaving the P2P protocol stuck part way through a `PUT`, a synthetic empty `DATA` followed by `INVALID` will be used to get the P2P protocol back into a state where it will accept any request.] ## parts of P2P protocol that are not supported over HTTP `NOTIFYCHANGE` is not supported, but it would be possible to extend this HTTP protocol to support it. `CONNECT` is not supported, and due to the bi-directional message passing nature of it, it cannot easily be done over HTTP (would need websockets). It should not be necessary anyway, because the git repository itself can be accessed over HTTP.