Commit graph

45321 commits

Author SHA1 Message Date
Joey Hess
74c6175795
fix serveGet early handle close
Needed that waitv after all..
2024-07-11 09:55:17 -04:00
Joey Hess
2c13e6c165
fix annexworker shutdown on early client disconnect 2024-07-11 09:15:52 -04:00
Joey Hess
6e3d35744d
cleanup 2024-07-11 07:56:55 -04:00
Joey Hess
3b37b9e53f
fix serveGet hang
This came down to SendBytes waiting on the waitv. Nothing ever filled
it.

Only Annex.Proxy needs the waitv, and it handles filling it. So make it
optional.
2024-07-11 07:46:52 -04:00
benjamin.poldrack@d09ccff6d42dd20277610b59867cf7462927b8e3
a82a573f75 2024-07-11 07:47:27 +00:00
benjamin.poldrack@d09ccff6d42dd20277610b59867cf7462927b8e3
9ce207532e 2024-07-11 07:23:30 +00:00
Joey Hess
80fb5445b5
a little progress on serveGet hang
Now it gets to the validity checker, but it seems it never runs it.
2024-07-10 17:48:48 -04:00
Joey Hess
8cb1332407
update 2024-07-10 16:10:08 -04:00
Joey Hess
1e0f92a5a1
implemented serveGet and clientGet
Both are only at bare proof of concept stage. Still need to deal with
signaling validity and invalidity, and checking it.

And there's a bad bug: After -JN*2 requests, another request hangs!

So, I think it's failing to free up the Annex worker and end of request
lifetime.

Perhaps I need to use this:

https://docs.servant.dev/en/stable/cookbook/managed-resource/ManagedResource.html
2024-07-10 16:06:39 -04:00
Joey Hess
f9b7ce7224
add Annex worker pool to P2PHttp
This will be needed for get and store, since those need to run Annex
actions.

withLocalP2PConnections will also probably use it.
2024-07-10 12:19:47 -04:00
Joey Hess
d4b9aea87b
implement gettimestamp 2024-07-10 10:23:10 -04:00
Joey Hess
7c588a5791
implement remove-before
The reason to use removeBeforeRemoteEndTime is twofold.

First, removeBefore sends two protocol commands. Currently, the HTTP
protocol runner only supports sending a single command per invocation.

Secondly, the http server gets a monotonic timestamp from the client. So
translating back to a POSIXTime would be annoying.

The timestamp flow with a proxy will be:

- client gets timestamp, which gets the monotonic timestamp from the
  proxied remote via the proxy. The timestamp is currently not
  proxied when there is a single proxy.
- client calls remove-before
- http server calls removeBeforeRemoteEndTime which sends REMOVE-BEFORE
  to the proxied remote.
2024-07-10 10:03:26 -04:00
Joey Hess
e9cba0a580
Revert "proxy local timestamps in single proxy case as well as cluster case"
Turns out not to be necessary. I think.

This reverts commit 81e11efda1.
2024-07-10 09:45:23 -04:00
Joey Hess
81e11efda1
proxy local timestamps in single proxy case as well as cluster case
This is less efficient, but it guarantees that the timestamps that the
client is sending are local timestamps, which turns out to be necessary
for the HTTP PTP protocol server.
2024-07-10 09:40:13 -04:00
Joey Hess
518248f009
remove debug print 2024-07-10 09:37:43 -04:00
Joey Hess
b8a26712c6
implement clientRemove
Tested removal.
2024-07-10 09:20:13 -04:00
Joey Hess
48f76cb3e8
implement serveRemove and send WWW-Authenticate header on auth failure 2024-07-10 09:13:01 -04:00
Joey Hess
97d0fc9b65
git-annex p2phttp options 2024-07-10 00:01:55 -04:00
Joey Hess
6a8a4d1775
authentication is implemented
just need to make Command.P2PHttp generate a GetServerMode from options
2024-07-09 20:54:47 -04:00
Joey Hess
e5bf49b879
http basic authorization header parsing
Sadly servant does not expose this though it also implements it.
2024-07-09 20:07:53 -04:00
Joey Hess
08371c3745
started on auth 2024-07-09 17:30:55 -04:00
Joey Hess
b5b3d8cde2
update 2024-07-09 14:30:50 -04:00
Joey Hess
dcd77ee555
fix p2phttp server to not get stuck
Process 1 command, then stop. Hopefully each of the Handlers will only
need 1 command.
2024-07-09 14:26:30 -04:00
Joey Hess
3d13521479
set up handles for p2phttp
Now it fully works.. for the first request. But then it gets stuck
waiting for the P2P protocol runner to shut down.
2024-07-09 13:50:42 -04:00
Joey Hess
edf8a3df2d
p2phttp is almost working for checkpresent
The server is fully running annex actions, only the P2PConnection is
wrong, currently using stdio.
2024-07-09 13:37:55 -04:00
Joey Hess
a3dd8b4bcb
capture API version in routes
Needed so the client can send it.
2024-07-09 12:04:29 -04:00
Joey Hess
751b8e0baf
implemented serveCheckPresent
Still need a way to run Proto though
2024-07-09 09:08:42 -04:00
yarikoptic
fade907c6a initial report from boox installation 2024-07-09 02:44:51 +00:00
Joey Hess
9a592f946f
split module 2024-07-08 21:12:23 -04:00
Joey Hess
3f402a20a8
implement Locker 2024-07-08 21:00:10 -04:00
Joey Hess
b758b01692
add lockids to http p2p protocol 2024-07-08 20:18:55 -04:00
Joey Hess
58031455dc
add lock map 2024-07-08 14:20:30 -04:00
Joey Hess
0bdee626ad
thread in a state 2024-07-08 14:00:23 -04:00
Joey Hess
69c4f07ab0
finish get API 2024-07-08 13:27:50 -04:00
Joey Hess
82d66ede5e
convert lockcontent api to http long polling
Websockets would work, but the problem with using them for this is that
each lockcontent call is a separate websocket connection. And that's an
actual TCP connection. One TCP connection per file dropped would be too
expensive. With http long polling, regular http pipelining can be used,
so it will reuse a TCP connection.

Unfortunately, at least with servant, bi-directional streams with long
polling don't result in true bidirectional full duplex communication.
Servant processes the whole client body stream before generating the server
body stream. I think it's entirely possible to do full bi-directional
communication over http, but it would need changes to servant.

And, there's no way for the client to tell if the server successfully
locked the content, since the server will keep processing the client
stream no matter what.:

So, added a new api endpoint, keeplocked. lockcontent will lock the key
for 10 minutes with retention lock, and then a call to keeplocked will
keep it locked for as long as needed. This does mean that there will
need to be a Map of locks by key, and I will probably want to add
some kind of lock identifier that lockcontent returns.
2024-07-08 12:57:46 -04:00
Joey Hess
522700d1c4
implemented servant-client support for websockets 2024-07-08 07:44:59 -04:00
Joey Hess
392b15d5c3
may have found a way to make a request for a websocket?!
dunno, it compiles anyway
2024-07-07 21:51:30 -04:00
Joey Hess
9ee005e49a
dummy HasClient ClientM WebSocket
Enough to let lockcontent routes be included and servant-client be used.
But not enough to use servant-client with those routes. May need to
implement a separate runner for that part of the protocol?

Also some misc other stuff needed to use servant-client.

And fix exposing of UUID in the JSON types. UUID does actually have
aeson instances, but they're used elsewhere (metadata --batch, although
only included to get it to compile, not actually used in there) and not
suitable for use here since this must work with every possible UUID.
2024-07-07 21:21:45 -04:00
Joey Hess
838169ee86
status 2024-07-07 16:16:11 -04:00
Joey Hess
bfa8c39adb
servant client mostly implemented
lockcontent had to be disabled until I can implement HasClient ClientM WebSocket

and in clientGet, it's not clear how to use the v1 and v0 versions,
which don't have a DataLengthHeader
2024-07-07 16:08:05 -04:00
Joey Hess
9a726cedf6
servant server now compiling
Just need to fill in some undefined
2024-07-07 14:48:20 -04:00
Joey Hess
1dbb5ec70d
servant API type is complete 2024-07-07 12:59:12 -04:00
Joey Hess
4133063ab1
Merge branch 'master' into httpproto 2024-07-07 12:08:24 -04:00
Joey Hess
86ce3bf1e4
started servant implementation of HTTP P2P protocol 2024-07-07 12:08:10 -04:00
Joey Hess
9595f77584
Merge branch 'master' of ssh://git-annex.branchable.com 2024-07-05 15:37:43 -04:00
Joey Hess
40306d3fcf
finalizing HTTP P2p protocol some more
Added v2-v0 endpoints. These are tedious, but will be needed in order to
use the HTTP protocol to proxy to repositories with older git-annex,
where git-annex-shell will be speaking an older version of the protocol.

Changed GET to use 422 when the content is not present. 404 is needed to
detect when a protocol version is not supported.
2024-07-05 15:34:58 -04:00
Joey Hess
fe0be28f68
playing around with bytestring lazy streaming in servant
Will be needed for git-annex http p2p protocol.
2024-07-05 15:34:10 -04:00
Joey Hess
2fb3ef4d41
finalizing HTTP P2P protocol
Managed to avoid netstrings. Actually, using netstrings while streaming
lazy ByteString turns out to be very difficult. So instead, have a
header that specifies the expected amount of data, and then it can just
arrange to send a different amount of data if it needs to indicate
INVALID.

Also improved the interface for GET of a key.
2024-07-05 15:03:51 -04:00
Joey Hess
5e564947d7
use netstrings for framing binary data with json at the end
This will be easy to implement with servant. It's also very efficient,
and fairly future-proof. Eg, could add another frame with other data.

This does make it a bit harder to use this protocol, but netstrings
probably take about 5 minutes to implement? Let's see...

import Text.Read
import Data.List

toNetString :: String -> String
toNetString s = show (length s) ++ ":" ++ s ++ ","

nextNetString :: String -> Maybe (String, String)
nextNetString s = case break (== ':') s of
        ([], _) -> Nothing
        (sn, rest) -> do
                n <- readMaybe sn
                let (v, rest') = splitAt n (drop 1 rest)
                return (v, drop 1 rest')

Ok, well, that took about 10 minutes ;-)
2024-07-05 11:53:03 -04:00
Joey Hess
95ba4d4480
thoughts on CGI, and use json 2024-07-05 10:08:43 -04:00