c81768d425
Unfortunately ReceiveMessage didn't handle unknown messages the way it was documented to; client sending VERSION would cause the server to return an ERROR and hang up. Fixed that, but old releases of git-annex use the P2P protocol for tor and will still have that behavior. So, version is not negotiated for Remote.P2P connections, only for Remote.Git connections, which will support VERSION from their first release. There will need to be a later flag day to change Remote.P2P; left a commented out line that is the only thing that will need to be changed then. Version 1 of the P2P protocol is not implemented yet, but updated the docs for the DATA change that will be allowed by that version. This commit was sponsored by Jeff Goeke-Smith on Patreon.
207 lines
6.1 KiB
Markdown
207 lines
6.1 KiB
Markdown
The git-annex P2P protocol is a custom protocol that git-annex uses to
|
|
communicate between peers.
|
|
|
|
There's a common line-based serialization of the protocol, but other
|
|
serializations are also possible. The line-based serialization is spoken
|
|
by [[git-annex-shell], and by git-annex over tor.
|
|
|
|
One peer is known as the client, and is the peer that initiates the
|
|
connection and sends commands. The other peer is known as the server, and
|
|
is the peer that the client connects to. It's possible for two connections
|
|
to be run at the same time between the same two peers, in different
|
|
directions.
|
|
|
|
## Errors
|
|
|
|
Either the client or the server may send an error message at any
|
|
time.
|
|
|
|
When the client sends an ERROR, the server will close the connection.
|
|
|
|
If the server sends an ERROR in response to the client's
|
|
request, the connection will remain open, and the client can make
|
|
another request.
|
|
|
|
ERROR this repository is read-only; write access denied
|
|
|
|
## Authentication
|
|
|
|
The protocol genernally starts with authentication. However, if
|
|
authentication already occurs on another layer, as is the case with
|
|
git-annex-shell, authentication will be skipped.
|
|
|
|
The client starts by sending an authentication command to the server,
|
|
along with its UUID. The AuthToken is some arbitrary token that has been
|
|
agreed upon beforehand.
|
|
|
|
AUTH UUID AuthToken
|
|
|
|
The server responds with either its own UUID when authentication
|
|
is successful. Or, it can fail the authentication, and close the
|
|
connection.
|
|
|
|
AUTH_SUCCESS UUID
|
|
AUTH_FAILURE
|
|
|
|
Note that authentication does not guarantee that the client is talking to
|
|
who they expect to be talking to. This, and encryption of the connection,
|
|
are handled at a lower level.
|
|
|
|
## Protocol version
|
|
|
|
The default protocol version is 0. The client can choose to
|
|
negotiate a new version with the server. This must come after
|
|
any authentication.
|
|
|
|
The client sends the highest protocol version it supports:
|
|
|
|
VERSION 2
|
|
|
|
The server responds with the highest protocol version it supports
|
|
that is less than or equal to the version the client sent:
|
|
|
|
VERSION 1
|
|
|
|
Now both client and server should use version 1.
|
|
|
|
(Note that old versions of git-annex, which speak the P2P protocol
|
|
over tor, don't support this, and attempting to negotiate a version
|
|
will cause the server to hang up the connection. To deal with this
|
|
historical bug, the version is not currently negotiated when using the
|
|
protocol over tor. At some point in the future, when all peers can be
|
|
assumed to be upgraded, this will be changed.)
|
|
|
|
## Binary data
|
|
|
|
The protocol allows raw binary data to be sent. This is done
|
|
using a DATA message. In the line-based serialization, this comes
|
|
on its own line, followed by a newline and the binary data.
|
|
The Len value tells how many bytes of data to read.
|
|
|
|
DATA 3
|
|
foo1
|
|
|
|
Note that there is no newline after the binary data; the next protocol
|
|
message will come immediately after it.
|
|
|
|
In protocol version 1 and higher, the binary data is suffixed with one
|
|
additional byte. Normally that is "1". "0" is used to indicate when the
|
|
file being transferred changed content while it was being sent
|
|
(eg, it was unlocked and got edited). In protocol version 0, this
|
|
additional byte is not sent.
|
|
|
|
If the sender finds itself unable to send as many bytes of data as it
|
|
promised (perhaps because a file got truncated while it was being sent),
|
|
its only option is to close the protocol connection.
|
|
|
|
## Checking if content is present
|
|
|
|
To check if a key is currently present on the server, the client sends:
|
|
|
|
CHECKPRESENT Key
|
|
|
|
The server responds with either SUCCESS or FAILURE.
|
|
|
|
## Locking content
|
|
|
|
To lock content on the server, preventing it from being removed,
|
|
the client sends:
|
|
|
|
LOCKCONTENT Key
|
|
|
|
The server responds with either SUCCESS or FAILURE.
|
|
The former indicates the content is locked. It will remain
|
|
locked until the connection is broken, or the client
|
|
sends:
|
|
|
|
UNLOCKCONTENT Key
|
|
|
|
The server makes no response to that.
|
|
|
|
## Removing content
|
|
|
|
To remove a key's content from the server, the client sends:
|
|
|
|
REMOVE Key
|
|
|
|
The server responds with either SUCCESS or FAILURE.
|
|
|
|
## Storing content on the server
|
|
|
|
To store content on the server, the client sends:
|
|
|
|
PUT AssociatedFile Key
|
|
|
|
Here AssociatedFile may be the name of a file in the git
|
|
repository, for information purposes only. Or it can be the
|
|
empty string. It will always have unix directory separators.
|
|
|
|
(Note that in the line-based serialization. AssociatedFile may not contain any
|
|
spaces, since it's not the last token in the line. Use '%' to indicate
|
|
whitespace.)
|
|
|
|
The server may respond with ALREADY-HAVE if it already
|
|
had the conent of that key. Otherwise, it responds with:
|
|
|
|
PUT-FROM Offset
|
|
|
|
Offset is the number of bytes into the file that the server wants
|
|
the client to start. This allows resuming transfers.
|
|
|
|
The client then sends a DATA message with content of the file from
|
|
the offset to the end of file.
|
|
|
|
If the server successfully receives the data and stores the content,
|
|
it replies with SUCCESS. Otherwise, FAILURE.
|
|
|
|
## Getting content from the server
|
|
|
|
To get content from the server, the client sends:
|
|
|
|
GET Offset AssociatedFile Key
|
|
|
|
The Offset is the number of bytes into the file that the client wants
|
|
the server to skip, which allows resuming transfers.
|
|
See description of AssociatedFile above.
|
|
|
|
The server then sends a DATA message with the content of the file
|
|
from the offset to end of file.
|
|
|
|
The client replies with SUCCESS or FAILURE.
|
|
|
|
## Connection to services
|
|
|
|
This is used to connect to services like git-upload-pack and
|
|
git-receive-pack that speak their own protocol.
|
|
|
|
The client sends a message to request the connection.
|
|
Service is the name of the service, eg "git-upload-pack".
|
|
|
|
CONNECT Service
|
|
|
|
Both client and server may now exchange DATA messages in any order,
|
|
encapsulating the service's protocol.
|
|
|
|
When the service exits, the server indicates this by telling the client
|
|
its exit code.
|
|
|
|
CONNECTDONE ExitCode
|
|
|
|
## Change notification
|
|
|
|
The client can request to be notified when a ref in
|
|
the git repository on the server changes.
|
|
|
|
NOTIFYCHANGE
|
|
|
|
The server will block until at least
|
|
one of the refs changes, and send a list of changed
|
|
refs.
|
|
|
|
CHANGED ChangedRefs
|
|
|
|
For example:
|
|
|
|
CHANGED refs/heads/master refs/heads/git-annex
|
|
|
|
Some servers may not support this command.
|