drafting P2P protocol over http
This commit is contained in:
Joey Hess
parent
623f483a68
commit
069b976698
3 changed files with 301 additions and 5 deletions
|
|
@ -201,6 +201,9 @@ was being sent.
|
|||
|
||||
The client replies with SUCCESS or FAILURE.
|
||||
|
||||
Note that the client responding with SUCCESS does not indicate to the
|
||||
server that it has stored the content. It may receive it and throw it away.
|
||||
|
||||
## Connection to services
|
||||
|
||||
This is used to connect to services like git-upload-pack and
|
||||
|
|
|
|||
|
|
@ -72,20 +72,31 @@ needs to know that a series of requests are part of the same P2P protocol
|
|||
session. In the example above, it would not have a good way to do that.
|
||||
One solution would be to add a session identifier UUID to each request.
|
||||
|
||||
## approach 2: HTTP API
|
||||
## approach 2: websockets
|
||||
|
||||
The client connects to the server over a websocket. From there on,
|
||||
the protocol is encapsulated in websockets.
|
||||
|
||||
This seems nice and simple, but again not very web native.
|
||||
|
||||
Some requests like `LOCKCONTENT` seem likely to need full duplex
|
||||
communication like websockets provide. But, it might be more web native to
|
||||
only use websockets for that request, and not for everything.
|
||||
|
||||
## approach 3: HTTP API
|
||||
|
||||
Another approach is to define a web-native API with endpoints that
|
||||
correspond to each action in the P2P protocol.
|
||||
|
||||
Something like this:
|
||||
|
||||
> GET /git-annex/v1/AUTH?clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925 HTTP/1.0
|
||||
> POST /git-annex/v1/AUTH?clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925 HTTP/1.0
|
||||
< AUTH-SUCCESS ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6
|
||||
|
||||
> GET /git-annex/v1/CHECKPRESENT?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
|
||||
> POST /git-annex/v1/CHECKPRESENT?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
|
||||
> SUCCESS
|
||||
|
||||
> GET /git-annex/v1/PUT-FROM?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
|
||||
> POST /git-annex/v1/PUT-FROM?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
|
||||
< PUT-FROM 0
|
||||
|
||||
> POST /git-annex/v1/PUT?key=SHA1--foo&associatedfile=bar&put-from=0&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
|
||||
|
|
@ -102,6 +113,8 @@ This needs a more complex spec. But it's easier for others to implement,
|
|||
especially since it does not need a session identifier, so the HTTP server can
|
||||
be stateless.
|
||||
|
||||
A full draft protocol for this is being developed at [[p2p_protocol_over_http/draft1]].
|
||||
|
||||
## HTTP GET
|
||||
|
||||
It should be possible to support a regular HTTP get of a key, with
|
||||
|
|
@ -122,4 +135,11 @@ The CONNECT message allows both sides of the P2P protocol to send DATA
|
|||
messages in any order. This seems difficult to encapsulate in HTTP.
|
||||
|
||||
Probably this can be not implemented, it's probably not needed for a HTTP
|
||||
remote?
|
||||
remote? This is used to tunnel git protocol over the P2P protocol, but for
|
||||
a HTTP remote the git repository can be accessed over HTTP as well.
|
||||
|
||||
## security
|
||||
|
||||
Should support HTTPS and/or be limited to only HTTPS.
|
||||
|
||||
Authentication via http basic auth?
|
||||
|
|
|
|||
273
doc/design/p2p_protocol_over_http/draft1.mdwn
Normal file
273
doc/design/p2p_protocol_over_http/draft1.mdwn
Normal file
|
|
@ -0,0 +1,273 @@
|
|||
[[!toc ]]
|
||||
|
||||
Draft 1 of a complete [[P2P_protocol]] over HTTP.
|
||||
|
||||
## git-annex protocol endpoint and version
|
||||
|
||||
The git-annex protocol endpoint is "/git-annex" appended to the HTTP
|
||||
url of a git remote.
|
||||
|
||||
## authentication
|
||||
|
||||
A git-annex protocol endpoint can optionally operate in readonly mode without
|
||||
authentication.
|
||||
|
||||
Authentication is required to make any changes.
|
||||
|
||||
Authentication is done using HTTP basic auth.
|
||||
|
||||
The user is recommended to only authenticate over HTTPS, since otherwise
|
||||
HTTP basic auth (as well as git-annex data) can be snooped. But some users
|
||||
may want git-annex to use HTTP in eg a LAN.
|
||||
|
||||
## protocol version
|
||||
|
||||
Each request in the protocol is versioned. The versions correspond
|
||||
to P2P protocol versions, but for simplicity, the minimum version supported
|
||||
over HTTP is version 2. Every implementation of the HTTP protocol must
|
||||
support version 2.
|
||||
|
||||
The protocol version comes before the request. Eg: `/git-annex/v2/put`
|
||||
|
||||
If the server does not support a particular protocol version, the
|
||||
request will fail with a 404, and the client should fall back to an earlier
|
||||
protocol version, eg version 2.
|
||||
|
||||
## common request parameters
|
||||
|
||||
Every request has some common parameters that are always included:
|
||||
|
||||
* `clientuuid`
|
||||
|
||||
The value is the UUID of the git-annex repository of the client.
|
||||
|
||||
* `serveruuid`
|
||||
|
||||
The value is the UUID of the git-annex repository that the server
|
||||
should serve.
|
||||
|
||||
Any request may also optionally include these parameters:
|
||||
|
||||
* `bypass`
|
||||
|
||||
The value is the UUID of a cluster gateway, which the server should avoid
|
||||
connecting to when serving a cluster. This is the equivilant of the
|
||||
`BYPASS` message in the [[P2P_Protocol]].
|
||||
|
||||
This parameter can be given multiple times to list several cluster
|
||||
gateway UUIDs.
|
||||
|
||||
[Internally, git-annex can use these common parameters, plus the protocol
|
||||
version, to create a P2P session. The P2P session is driven through
|
||||
the AUTH, VERSION, and BYPASS messages, leaving the session ready to
|
||||
service requests.]
|
||||
|
||||
## request messages
|
||||
|
||||
All the requests below are sent with the HTTP POST method.
|
||||
|
||||
### checkpresent
|
||||
|
||||
Checks if a key is currently present on the server.
|
||||
|
||||
Example:
|
||||
|
||||
> POST /git-annex/v2/checkpresent?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
||||
< SUCCESS
|
||||
|
||||
There is one required additional parameter, `key`.
|
||||
|
||||
The body of the request is empty.
|
||||
|
||||
The server responds with "SUCCESS" if the key is present
|
||||
or "FAILURE" if it is not present.
|
||||
|
||||
### lockcontent
|
||||
|
||||
Locks the content of a key on the server, preventing it from being removed.
|
||||
|
||||
There is one required additional parameter, `key`.
|
||||
|
||||
This request opens a websocket between the client and the server.
|
||||
The server sends "SUCCESS" over the websocket once it has locked
|
||||
the content. Or it sends "FAILURE" if it is unable to lock the content.
|
||||
|
||||
Once the server has sent "SUCCESS", the content remains locked as long as
|
||||
the client remains connected to the websocket. When the client disconnects,
|
||||
or closes the websocket, the server unlocks the content.
|
||||
|
||||
XXX What happens if the connection times out? Will the client notice that
|
||||
in time? How does this work with P2P over ssh?
|
||||
|
||||
### remove
|
||||
|
||||
Remove a key's content from the server.
|
||||
|
||||
Example:
|
||||
|
||||
> POST /git-annex/v2/remove?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
||||
< SUCCESS
|
||||
|
||||
There is one required additional parameter, `key`.
|
||||
|
||||
The body of the request is empty.
|
||||
|
||||
The server responds with "SUCCESS" if the key was removed,
|
||||
or "FAILURE" if the key was not able to be removed.
|
||||
|
||||
The server can also respond with "SUCCESS-PLUS" or "FAILURE-PLUS".
|
||||
Each has a subsequent list of UUIDs of repositories
|
||||
that the content was removed from. For example:
|
||||
|
||||
SUCCESS-PLUS 702ce472-38a1-11ef-864f-23851a2edf71 707dea20-38a1-11ef-96a4-fb7e8c8369f0
|
||||
|
||||
If the server was prevented from trying to remove the key due to a policy
|
||||
(eg due to being read-only or append-only, it will respond with "ERROR",
|
||||
followed by a space and an error message.
|
||||
|
||||
### put
|
||||
|
||||
Store content on the server.
|
||||
|
||||
Example:
|
||||
|
||||
> POST /git-annex/v2/put?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
||||
> Content-Type: application/octet-stream
|
||||
> Content-Length: 4
|
||||
> foo1
|
||||
< SUCCESS
|
||||
|
||||
There is one required additional parameter, `key`.
|
||||
|
||||
There is are also these optional parameters:
|
||||
|
||||
* `associatedfile`
|
||||
|
||||
The name of a file in the git repository, for informational purposes
|
||||
only.
|
||||
|
||||
* `offset`
|
||||
|
||||
Number of bytes that have been omitted from the beginning of the file.
|
||||
Usually this will be determined by making a `putoffset` request.
|
||||
|
||||
The body of the request is the content of the key, starting from the
|
||||
specified offset or from the beginning. After the content of the key,
|
||||
there is one more byte.
|
||||
|
||||
The additional byte is "1" to indicate that the content was not changed
|
||||
while it was being sent, or "0" to indicate that modified content was sent
|
||||
and should be disregarded by the server. (This corresponds
|
||||
to the `VALID` and `INVALID` messages in the P2P protocol.)
|
||||
|
||||
The `Content-Type` header should be `application/octet-stream`.
|
||||
|
||||
The `Content-Length` header should be set to the length of the body.
|
||||
|
||||
The server responds with `SUCCESS` if it received the data and stored the
|
||||
content. If it was unable to do so, it responds with `FAILURE`.
|
||||
|
||||
The server can also reply with `SUCCESS-PLUS`, which has a subsequent list of
|
||||
UUIDs of repositories that the content was stored to. For example:
|
||||
|
||||
SUCCESS-PLUS 702ce472-38a1-11ef-864f-23851a2edf71 707dea20-38a1-11ef-96a4-fb7e8c8369f0
|
||||
|
||||
If the server was prevented from storing the key due to a policy
|
||||
(eg due to being read-only), it will respond with "ERROR", followed
|
||||
by a space and an error message.
|
||||
|
||||
### putoffset
|
||||
|
||||
Asks the server what `offset` can be used in a `put` of a key.
|
||||
|
||||
This should usually be used right before sending a `put` request.
|
||||
The offset may not be valid after some point in time, which could result in
|
||||
the `put` request failing.
|
||||
|
||||
Example:
|
||||
|
||||
> POST /git-annex/v2/putoffset?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
||||
< 10
|
||||
|
||||
There is one required additional parameter, `key`.
|
||||
|
||||
The body of the request is empty.
|
||||
|
||||
The server responds with the largest allowable offset.
|
||||
|
||||
If the server was prevented from storing the key due to a policy
|
||||
(eg due to being read-only), it will respond with "ERROR", followed
|
||||
by a space and an error message.
|
||||
|
||||
[Implementation note: This will be implemented by sending `PUT` and
|
||||
returning the `PUT-FROM` offset. To avoid leaving the P2P protocol stuck
|
||||
part way through a `PUT`, a synthetic empty `DATA` followed by `INVALID`
|
||||
will be used to get the P2P protocol back into a state where it will accept
|
||||
any request.]
|
||||
|
||||
### get
|
||||
|
||||
Get content from the server.
|
||||
|
||||
Example:
|
||||
|
||||
> POST /git-annex/v2/get?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
|
||||
< Content-Type: application/octet-stream
|
||||
< Content-Length: 4
|
||||
< foo1
|
||||
|
||||
There is one required additional parameter, `key`.
|
||||
|
||||
There is are also these optional parameters:
|
||||
|
||||
* `associatedfile`
|
||||
|
||||
The name of a file in the git repository, for informational purposes
|
||||
only.
|
||||
|
||||
* `offset`
|
||||
|
||||
Number of bytes to skip sending from the beginning of the file.
|
||||
|
||||
The body of the request is empty.
|
||||
|
||||
The server's response will have a `Content-Type` header of
|
||||
`application/octet-stream`.
|
||||
|
||||
The server's response will have a `Content-Length` header
|
||||
set to the length of the body.
|
||||
|
||||
The server's response body is the content of the key, from the specified
|
||||
offset. After the content of the key, there is one more byte.
|
||||
|
||||
The additional byte is "1" to indicate that the content was not changed
|
||||
while it was being sent, or "0" to indicate that modified content was sent
|
||||
and should be discarded by the client. (This corresponds
|
||||
to the `VALID` and `INVALID` messages in the P2P protocol.)
|
||||
|
||||
Note that, if the server is not able to send the content of the requested
|
||||
key, its response body will consist of "0", eg 0 bytes of content which is
|
||||
not valid. On the other hand, a response body of "1" is used for an empty
|
||||
key which is valid.
|
||||
|
||||
## simple HTTP GET
|
||||
|
||||
The git-annex protocol endpoint also supports a regular HTTP get
|
||||
of a key. This is not part of the P2P protocol, but is provided for
|
||||
convenience, to allow other clients than git-annex to easily download
|
||||
the content of a key.
|
||||
|
||||
> GET /git-annex/key/SHA1--foo HTTP/1.1
|
||||
< Content-Type: application/octet-stream
|
||||
< Content-Length: 3
|
||||
< foo
|
||||
|
||||
## parts of P2P protocol that are not supported over HTTP
|
||||
|
||||
`NOTIFYCHANGE` is not supported, but it would be possible to extend
|
||||
this HTTP protocol to support it.
|
||||
|
||||
`CONNECT` is not supported, and due to the bi-directional message passing
|
||||
nature of it, it cannot easily be done over HTTP. It should not be
|
||||
necessary anyway, because the git repository itself can be accessed over
|
||||
HTTP.
|
||||
Loading…
Reference in a new issue