fix hang at end of PUT to proxied p2p http remote

sendExactly will now be sure to evaluate the whole lazy ByteString.

In this case, the lazy ByteString was exactly the right lenth.
But, it seems that L.take caused it to not actually be fully evaluated.

In servePut, this manifested as gather never being fully evaluated,
which caused the hang.

Very, very subtle, and horrible bug. Clearly the use of lazy ByteString
(or really just laziness) is at fault, and it would be very worth moving
to conduit or whatever to avoid this.
This commit is contained in:
Joey Hess 2024-07-26 19:50:15 -04:00
parent b431201e1f
commit a3dab58be2
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
3 changed files with 19 additions and 6 deletions

View file

@ -335,12 +335,16 @@ debugMessage conn prefix m = do
-- Must avoid sending too many bytes as it would confuse the other end. -- Must avoid sending too many bytes as it would confuse the other end.
-- This is easily dealt with by truncating it. -- This is easily dealt with by truncating it.
-- --
-- However, the whole ByteString will be evaluated here, even if
-- the end of it does not get sent.
--
-- If too few bytes are sent, the only option is to give up on this -- If too few bytes are sent, the only option is to give up on this
-- connection. False is returned to indicate this problem. -- connection. False is returned to indicate this problem.
sendExactly :: Len -> L.ByteString -> Handle -> MeterUpdate -> IO Bool sendExactly :: Len -> L.ByteString -> Handle -> MeterUpdate -> IO Bool
sendExactly (Len n) b h p = do sendExactly (Len n) b h p = do
sent <- meteredWrite' p (B.hPut h) (L.take (fromIntegral n) b) let (x, y) = L.splitAt (fromIntegral n) b
return (fromBytesProcessed sent == n) sent <- meteredWrite' p (B.hPut h) x
L.length y `seq` return (fromBytesProcessed sent == n)
receiveExactly :: Len -> Handle -> MeterUpdate -> IO L.ByteString receiveExactly :: Len -> Handle -> MeterUpdate -> IO L.ByteString
receiveExactly (Len n) h p = hGetMetered h (Just n) p receiveExactly (Len n) h p = hGetMetered h (Just n) p

View file

@ -115,7 +115,7 @@ the client sends:
The server responds with either SUCCESS or FAILURE. The server responds with either SUCCESS or FAILURE.
The former indicates the content is locked. It will remain The former indicates the content is locked. It will remain
locked until the client sends: locked until the client sends its next message, which must be:
UNLOCKCONTENT Key UNLOCKCONTENT Key

View file

@ -28,10 +28,19 @@ Planned schedule of work:
## work notes ## work notes
* http server proxying hangs on git-annex copy --to it * This against a http proxied remote leads to a protocol error:
All the data gets sent first. Suggests the hang is at connection teardown git-annex move foo --to origin-c
time. git-annex get foo --from origin-c
ERROR expected UNLOCKCONTENT
May need to run the commands a few times before it happens.
I think it's because proxyRequest treats LOCKCONTENT as a single
command+reponse, with UNLOCKCONTENT separately. So it's possible for
there to be two different connections to the proxied remote,
with LOCKCONTENT being sent to one, and UNLOCKCONTENT to the other one.
* test http server proxying with special remotes * test http server proxying with special remotes