Commit graph

45350 commits

Author SHA1 Message Date
Joey Hess
4304f1b6ae
better handling of content not available from cluster
Sending ERROR caused the client to get confused and protocol to freeze.
Better to send empty DATA and indicate it's not valid.

This fixes a hang in git-annex testremote of a cluster accessed via the
http server. That testremote is still failing, for some reason after
storing a test key, the cluster reports it as not present.
2024-07-28 11:09:07 -04:00
Joey Hess
fbbedae497
add --clusterjobs option and default to 1
The default of 1 is not ideal at all, but it avoids an accidental M*N
causing so much concurrency it becomes unusable.
2024-07-28 10:36:22 -04:00
Joey Hess
1259ad89b6
cluster support in http API server
Wired it up and it seems to basically work, although the test suite is
not fully passing.

Note that --jobs currently gets multiplied by the number of nodes in the
cluster, which is probably not good.
2024-07-28 10:17:29 -04:00
Joey Hess
8ec174408e
remove duplicate code 2024-07-28 09:35:09 -04:00
Joey Hess
0cdd418407
tested shutdown of connection to http proxied special remote
I had worried it might not work properly, but it does, the endv works.
2024-07-28 09:17:47 -04:00
Joey Hess
ef8f24f28c
fix PUT to http proxied special remote
It was hanging because it never sent FAILURE in the INVALID case.
And putoffset always triggers the INVALID case.
2024-07-28 09:14:42 -04:00
Joey Hess
0ea645944e
thoughts on exporttree 2024-07-27 19:59:54 -04:00
Joey Hess
1c0448e33c
update 2024-07-26 20:44:01 -04:00
Joey Hess
0fb86d2916
UNLOCKCONTENT is not a top-level request
proxyRequest was treating UNLOCKCONTENT as a separate request.
That made it possible for there to be two different connections to the
proxied remote, with LOCKCONTENT being sent to one, and UNLOCKCONTENT
to the other one. A protocol error.

git-annex testremote now passes against a http proxied remote.
2024-07-26 20:39:06 -04:00
Joey Hess
a3dab58be2
fix hang at end of PUT to proxied p2p http remote
sendExactly will now be sure to evaluate the whole lazy ByteString.

In this case, the lazy ByteString was exactly the right lenth.
But, it seems that L.take caused it to not actually be fully evaluated.

In servePut, this manifested as gather never being fully evaluated,
which caused the hang.

Very, very subtle, and horrible bug. Clearly the use of lazy ByteString
(or really just laziness) is at fault, and it would be very worth moving
to conduit or whatever to avoid this.
2024-07-26 19:50:15 -04:00
Joey Hess
b431201e1f
update 2024-07-26 17:15:09 -04:00
Joey Hess
d1faa13d6a
implement proxy connection pool
removeOldestProxyConnectionPool will be innefficient the larger the pool
is. A better data structure could be more efficient. Eg, make each value
in the pool include the timestamp of its oldest element, then the oldest
value can be found and modified, rather than rebuilding the whole Map.

But, for pools of a few hundred items, this should be fine. It's O(n*n log n)
or so.

Also, when more than 1 connection with the same pool key exists,
it's efficient even for larger pools, since removeOldestProxyConnectionPool
is not needed.

The default of 1 idle connection could perhaps be larger.. like the
number of jobs? Otoh, it seems good to ramp up and down the number of
connections, which does happen. With 1, there is at most one stale
connection, which might cause a request to fail.
2024-07-26 17:03:31 -04:00
yarikoptic
de90a2c5de initial report on keeping association with the remote 2024-07-26 20:01:23 +00:00
Joey Hess
fb43b7ea3f
closeP2PConnection on interrupted GET 2024-07-26 15:50:01 -04:00
Joey Hess
267a202e72
clean up after http p2p proxy GET is interrupted
There was an annex worker thread that did not get stopped.

It was stuck in ReceiveMessage from the P2PHandleTMVar.

Fixed by making P2PHandleTMVar closeable.

In serveGet, releaseP2PConnection has to come first, else the
annexworker may not shut down, if it's waiting to read from it.

In proxyConnection, call closeRemoteSide in order to wait for the ssh
process (for example).
2024-07-26 15:33:20 -04:00
Joey Hess
5ebbb31b36
close proxy remote side when done with it 2024-07-26 13:57:28 -04:00
Joey Hess
b028b1a379
oops 2024-07-26 13:55:14 -04:00
Joey Hess
212e5d04e6
oops 2024-07-26 13:54:37 -04:00
Joey Hess
ad025b8e5e
clean up protocol version for proxying
The proxy always checks the protocol version of a remote before talking
to it in a version-specific way, so the protocol version in the ProxyParams
is the client's protocol version. The remote will always be at the same or
an older protocol version than the client.

Note that in relayDATAFinish, when the client is at protocol version 0,
the remote must thus be as well, and that's why its version is not
checked in the case for that.

With that clarified, it's evident that, in P2P.Http.State, there's no
need to look at the proxied remote's protocol version at all.
2024-07-26 13:49:05 -04:00
Joey Hess
de7730f51f
fix setting protocol version for v0 server
Before it was leaving the protocol version at the default, when it
was actually v0.

v0 has only ever been used over tor, which is probably why this bug was
never noticed.
2024-07-26 13:24:56 -04:00
Joey Hess
576ec6ed71
fix hang in GET from http p2p proxy
serverP2PConnection = proxyfromclientconn causes serveGet to
signalFullyConsumedByteString to it, which is what it's waiting for
2024-07-26 12:51:00 -04:00
Joey Hess
60ccbc4fe6
improve comment 2024-07-26 12:48:47 -04:00
Joey Hess
f052091558
update 2024-07-26 11:01:45 -04:00
Joey Hess
cc1da2d516
http p2p proxy is now largely working 2024-07-26 10:44:10 -04:00
m.risse@77eac2c22d673d5f10305c0bade738ad74055f92
17f7912dd5 Added a comment 2024-07-26 11:42:49 +00:00
Joey Hess
b391756b32
remove some debugging 2024-07-25 21:36:10 -04:00
Joey Hess
96ad0ccc5b
wip 2024-07-25 15:39:57 -04:00
Joey Hess
6ef6ad808f
use a record to reduce the huge number of parameters 2024-07-25 15:23:18 -04:00
Joey Hess
7b56fe1350
factored out proxyRequest
This will allow the HTTP server to keep a connection to a proxied
remote open, and only enter the Annex monad when handling a client
request.
2024-07-25 15:00:36 -04:00
Joey Hess
3d14e2cf58
http server support for proxies, incomplete
Refactored git-annex-shell code so this can use checkCanProxy'.

At this point all that remains is opening a proxy connection,
and using a proxy connection.
2024-07-25 13:19:24 -04:00
Joey Hess
0bdeafc2c4
use annex+http for accessing proxies
Doesn't work yet on the http server side, which is throwing 502 bad
gateway.
2024-07-25 12:00:57 -04:00
Joey Hess
ba0ecbf47e
less indent 2024-07-25 10:12:59 -04:00
Joey Hess
b13c2407af
p2phttp drop supports checking proof timestamps
At this point the p2phttp implementation is fully complete!
2024-07-25 10:11:09 -04:00
Joey Hess
6a3f755bfa
add common parameters to generic get API
Honestly this was just done to make the documentation correct. There's
no point in using these parameters. And they're optional.
2024-07-24 20:55:58 -04:00
Joey Hess
ab22938c0b
fix build without servant 2024-07-24 15:13:02 -04:00
Joey Hess
515c42e1e3
testremote passes on p2phttp remote 2024-07-24 14:42:24 -04:00
Joey Hess
f5624a69e3
expire lock after 10 minutes initially
Once keeplocked is called, the lock will expire at the end
of that call. But if keeplocked never gets called, this avoids
the lock persisting forever.
2024-07-24 14:25:40 -04:00
Joey Hess
97836aafba
Remote.Git lockContent works with annex+http urls 2024-07-24 13:42:57 -04:00
Joey Hess
9fa9678585
Remote.Git removeKey works with annex+http urls
Does not yet handle drop proof lock timestamp checking.
2024-07-24 12:33:26 -04:00
Joey Hess
fd3bdb2300
update 2024-07-24 12:19:53 -04:00
Joey Hess
0d81d1ee2f
update 2024-07-24 12:18:51 -04:00
Joey Hess
cfdb80cd05
progress meter for p2phttp storeKey 2024-07-24 12:14:56 -04:00
Joey Hess
b3915b88ba
Remote.Git storeKey works with annex+http urls
Does not yet update progress meter.
2024-07-24 12:05:10 -04:00
Joey Hess
0280e2dd5e
update 2024-07-24 11:13:37 -04:00
Joey Hess
5b1ac1a313
more generic clientGet 2024-07-24 11:10:19 -04:00
Joey Hess
10f2c23fd7
fix slowloris timeout in hashing resume of download of large file
Hash the data that is already present in the file before connecting to
the http server.
2024-07-24 11:03:59 -04:00
Joey Hess
0594338a78
fix meter display on resume
It still needs to be offset, otherwise on resume from 80% it will
display 1%..20%.

Seems that this bug must have affected P2P.Annex as well where it runs
this code, but apparently it didn't affect it in a very user-visible
way. Maybe the transfer log file was updated incorrectly?
2024-07-24 10:28:48 -04:00
Joey Hess
7bd616e169
Remote.Git retrieveKeyFile works with annex+http urls
This includes a bugfix to serveGet, it hung at the end.
2024-07-24 10:28:44 -04:00
Joey Hess
a2d1844292
factor out resumeVerifyFromOffset 2024-07-24 09:29:44 -04:00
Joey Hess
ad945896c9
avoid needing ifdefs when using P2P.Http.Client 2024-07-24 08:33:59 -04:00