mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

use netstrings for framing binary data with json at the end

Browse source 
This will be easy to implement with servant. It's also very efficient,
and fairly future-proof. Eg, could add another frame with other data.

This does make it a bit harder to use this protocol, but netstrings
probably take about 5 minutes to implement? Let's see...

import Text.Read
import Data.List

toNetString :: String -> String
toNetString s = show (length s) ++ ":" ++ s ++ ","

nextNetString :: String -> Maybe (String, String)
nextNetString s = case break (== ':') s of
        ([], _) -> Nothing
        (sn, rest) -> do
                n <- readMaybe sn
                let (v, rest') = splitAt n (drop 1 rest)
                return (v, drop 1 rest')

Ok, well, that took about 10 minutes ;-)
This commit is contained in:
Joey Hess 2024-07-05 11:53:03 -04:00
parent 95ba4d4480
commit 5e564947d7
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
1 changed files with 36 additions and 15 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

51
doc/design/p2p_protocol_over_http/draft1.mdwn
View file

@ -62,6 +62,23 @@ version, to create a P2P session. The P2P session is driven through
the AUTH, VERSION, and BYPASS messages, leaving the session ready to
service requests.]
## binary data framing
When a request body or response body includes binary data, eg the content
of a large file, the body is framed using
[netstrings](http://cr.yp.to/proto/netstrings.txt).
The netstring framing is simply the length of the string in ASCII
digits, followed by the string, and then a comma.
This allows efficiently sending binary data in one frame, followed by a
second frame that can contain eg a JSON document.
For example, a body containing the binary data "foo" followed by
a JSON document `{"valid": true}` is framed like this:
3:foo,15:{"valid": true},
## request messages
All the requests below are sent with the HTTP POST method.
@ -166,9 +183,8 @@ Example:
> POST /git-annex/v3/put?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
> Content-Type: application/octet-stream
> Content-Length: 20
> foo
> {"valid": true}
> Content-Length: 25
> 3:foo,15:{"valid": true},
< {"stored": true}
There is one required additional parameter, `key`.
@ -185,14 +201,17 @@ There are are also these optional parameters:
Number of bytes that have been omitted from the beginning of the file.
Usually this will be determined by making a `putoffset` request.
The body of the request is the content of the key, starting from the
specified offset or from the beginning. After the content of the key,
there is a newline, followed by a JSON object.
The body of the request is two items framed with netstrings.
The JSON object has a field "valid" that is true when the content
was not changed while it was being sent, or false when modified
content was sent and should be disregarded by the server. (This corresponds
to the `VALID` and `INVALID` messages in the P2P protocol.)
The first item is the content of the key, starting from the specified
offset or from the beginning when no offset was specified.
The second item is a JSON object.
The JSON object has a field "valid" that is true when the content was not
changed while it was being sent, or false when modified content was sent
and should be disregarded by the server. (This corresponds to the `VALID`
and `INVALID` messages in the P2P protocol.)
The `Content-Type` header should be `application/octet-stream`.
@ -248,8 +267,7 @@ Example:
> POST /git-annex/v3/get?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
< Content-Type: application/octet-stream
> Content-Length: 20
> foo
> {"valid": true}
> 3:foo,15:{"valid": true},
There is one required additional parameter, `key`.
@ -272,9 +290,12 @@ The server's response will have a `Content-Type` header of
The server's response will have a `Content-Length` header
set to the length of the body.
The server's response body is the content of the key, from the specified
offset. After the content of the key, there is a newline, followed by a
JSON object.
The body of the response is two items framed with netstrings.
The first item is the content of the key, starting from the specified
offset or from the beginning when no offset was specified.
The second item is a JSON object.
The JSON object has a field "valid" that is true when the content
was not changed while it was being sent, or false when whatever