From 5e564947d7ce6b1dc18022b86cda55ff894e7070 Mon Sep 17 00:00:00 2001
From: Joey Hess <joeyh@joeyh.name>
Date: Fri, 5 Jul 2024 11:53:03 -0400
Subject: [PATCH] use netstrings for framing binary data with json at the end

This will be easy to implement with servant. It's also very efficient,
and fairly future-proof. Eg, could add another frame with other data.

This does make it a bit harder to use this protocol, but netstrings
probably take about 5 minutes to implement? Let's see...

import Text.Read
import Data.List

toNetString :: String -> String
toNetString s = show (length s) ++ ":" ++ s ++ ","

nextNetString :: String -> Maybe (String, String)
nextNetString s = case break (== ':') s of
        ([], _) -> Nothing
        (sn, rest) -> do
                n <- readMaybe sn
                let (v, rest') = splitAt n (drop 1 rest)
                return (v, drop 1 rest')

Ok, well, that took about 10 minutes ;-)
---
 doc/design/p2p_protocol_over_http/draft1.mdwn | 51 +++++++++++++------
 1 file changed, 36 insertions(+), 15 deletions(-)

diff --git a/doc/design/p2p_protocol_over_http/draft1.mdwn b/doc/design/p2p_protocol_over_http/draft1.mdwn
index 071321537b..73a621f404 100644
--- a/doc/design/p2p_protocol_over_http/draft1.mdwn
+++ b/doc/design/p2p_protocol_over_http/draft1.mdwn
@@ -62,6 +62,23 @@ version, to create a P2P session. The P2P session is driven through
 the AUTH, VERSION, and BYPASS messages, leaving the session ready to
 service requests.]
 
+## binary data framing
+
+When a request body or response body includes binary data, eg the content
+of a large file, the body is framed using 
+[netstrings](http://cr.yp.to/proto/netstrings.txt).
+
+The netstring framing is simply the length of the string in ASCII
+digits, followed by the string, and then a comma.
+
+This allows efficiently sending binary data in one frame, followed by a
+second frame that can contain eg a JSON document.
+
+For example, a body containing the binary data "foo" followed by
+a JSON document `{"valid": true}` is framed like this:
+    
+    3:foo,15:{"valid": true},
+
 ## request messages
 
 All the requests below are sent with the HTTP POST method.
@@ -166,9 +183,8 @@ Example:
 
     > POST /git-annex/v3/put?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
     > Content-Type: application/octet-stream
-    > Content-Length: 20
-    > foo
-    > {"valid": true}
+    > Content-Length: 25
+    > 3:foo,15:{"valid": true},
     < {"stored": true}
 
 There is one required additional parameter, `key`.
@@ -185,14 +201,17 @@ There are are also these optional parameters:
   Number of bytes that have been omitted from the beginning of the file. 
   Usually this will be determined by making a `putoffset` request.
 
-The body of the request is the content of the key, starting from the
-specified offset or from the beginning. After the content of the key,
-there is a newline, followed by a JSON object.
+The body of the request is two items framed with netstrings.
 
-The JSON object has a field "valid" that is true when the content 
-was not changed while it was being sent, or false when modified
-content was sent and should be disregarded by the server. (This corresponds
-to the `VALID` and `INVALID` messages in the P2P protocol.)
+The first item is the content of the key, starting from the specified
+offset or from the beginning when no offset was specified.
+
+The second item is a JSON object.
+
+The JSON object has a field "valid" that is true when the content was not
+changed while it was being sent, or false when modified content was sent
+and should be disregarded by the server. (This corresponds to the `VALID`
+and `INVALID` messages in the P2P protocol.)
 
 The `Content-Type` header should be `application/octet-stream`.
 
@@ -248,8 +267,7 @@ Example:
     > POST /git-annex/v3/get?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
     < Content-Type: application/octet-stream
     > Content-Length: 20
-    > foo
-    > {"valid": true}
+    > 3:foo,15:{"valid": true},
 
 There is one required additional parameter, `key`.
 
@@ -272,9 +290,12 @@ The server's response will have a `Content-Type` header of
 The server's response will have a `Content-Length` header 
 set to the length of the body.
 
-The server's response body is the content of the key, from the specified
-offset. After the content of the key, there is a newline, followed by a
-JSON object.
+The body of the response is two items framed with netstrings.
+
+The first item is the content of the key, starting from the specified
+offset or from the beginning when no offset was specified.
+
+The second item is a JSON object.
 
 The JSON object has a field "valid" that is true when the content 
 was not changed while it was being sent, or false when whatever