Commit graph

45159 commits

Author SHA1 Message Date
Joey Hess
75b1d50b99
add remoteAnnexP2PHttpUrl to RemoveGitConfig
This is always parsed, when building without servant, a Baseurl is not
generated, and users of it will need to fail.
2024-07-23 09:57:01 -04:00
Joey Hess
5c39652235
starting support for remote.name.annexUrl set to annex+http
In this case, Remote.Git should not use that url for all access to
the repository. It will only be used for annex operations, which isn't
done yet.
2024-07-23 09:12:21 -04:00
Joey Hess
fdb888a56a
update servant build flag
make it work when building w/o assistant
2024-07-23 08:53:56 -04:00
Joey Hess
6596fe80ec
update 2024-07-23 08:53:52 -04:00
Joey Hess
a6a03ca586
annex+http urls 2024-07-23 08:42:33 -04:00
Joey Hess
37373fad38
refactor 2024-07-22 21:19:32 -04:00
Joey Hess
2acde0152a
fix build 2024-07-22 21:19:20 -04:00
Joey Hess
758cff0fde
update 2024-07-22 20:59:45 -04:00
Joey Hess
06de2ad972
change default port to 9417
Port 80 would need root, not a good idea, so pick something that might
work by default.

9418 is git protocol's port. 9419 is used by something, but nothing
known uses 9417, so it's as good a default as any.
2024-07-22 20:52:17 -04:00
Joey Hess
53000cf06e
exit cleanly on eg, failure to bind socket 2024-07-22 20:37:37 -04:00
Joey Hess
9984252ab5
P2P protocol is finalized 2024-07-22 19:50:08 -04:00
Joey Hess
163a7e91c4
make keeplocked client UUID optional
It's not used, but it makes the protocol simpler if this parameter can
always be sent.
2024-07-22 19:48:54 -04:00
Joey Hess
7f4cff7ae9
locking over http basically working 2024-07-22 19:44:26 -04:00
Joey Hess
e979e85bff
make serveKeepLocked check auth just to be safe 2024-07-22 19:15:52 -04:00
Joey Hess
63e42aa1bc
implement clientLockContent 2024-07-22 18:55:12 -04:00
Joey Hess
f5dd7a8bc0
implemented serveLockContent (untested) 2024-07-22 17:38:42 -04:00
Joey Hess
d5eaf0f567
improve clientKeepLocked 2024-07-22 16:56:44 -04:00
Joey Hess
10682eb882
variable name 2024-07-22 16:50:11 -04:00
Joey Hess
8a48d08abd
refactor 2024-07-22 16:49:05 -04:00
Joey Hess
8d36e597f1
finished clientGet validity checking 2024-07-22 16:31:39 -04:00
Joey Hess
48eb6671e4
improve clientGet types 2024-07-22 16:23:08 -04:00
Joey Hess
b697c6b9da
fix TMVar left full crash affecting servePutOffset
Problem is that whatever is reading from the TMVar may not have read
from it yet before the client writes the next thing to it.
2024-07-22 15:48:46 -04:00
Joey Hess
3069e28dd8
implemented servePutOffset and clientPutOffset
But, it's buggy: the server hangs without processing the VALIDITY,
and I can't seem to work out why. As far as I can see, storefile
is getting as far as running the validitycheck, which is supposed to
read that, but never does.

This is especially strange because what seems like the same protocol
doesn't hang when servePut runs it. This made me think that it needed
to use inAnnexWorker to be more like servePut, but that didn't help.

Another small problem with this is that it does create an empty
.git/annex/tmp/ file for the key. Since this will usually be used in
combination with servePut, that doesn't seem worth worrying about much.
2024-07-22 15:04:10 -04:00
Joey Hess
c36b9cd1fc
fix crash in servePut when client sends from too high offset
Have to consume all the content from the client, but force an Invalid
result, so it will just get discarded.
2024-07-22 13:12:15 -04:00
Joey Hess
b240a11b79
clientPut seeking to offset 2024-07-22 12:50:21 -04:00
Joey Hess
a01426b713
avoid padding in servePut
This means that when the client sends a truncated data to indicate
invalidity, DATA is not passed the full expected data. That leaves the
P2P connection in a state where it cannot be reused. While so far, they
are not reused, they will be later when proxies are supported. So, have
to close the P2P connection in this situation.
2024-07-22 12:30:30 -04:00
Joey Hess
efa0efdc44
avoid padding in clientPut
Instead truncate when necessary to indicate invalid content was sent.
Very similar to how serveGet handles it.
2024-07-22 11:47:24 -04:00
Joey Hess
726c815a7f
fix releasing of p2p connection 2024-07-22 11:26:22 -04:00
Joey Hess
72d0769ca5
avoid padding content in serveGet
Always truncate instead. The padding risked something not noticing the
content was bad and getting a file that was corrupted in a novel way
with the padding "X" at the end. A truncated file is better.
2024-07-22 11:19:52 -04:00
Joey Hess
4826a3745d
servePut and clientPut implementation
Made the data-length header required even for v0. This simplifies the
implementation, and doesn't preclude extra verification being done for
v0.

The connectionWaitVar is an ugly hack. In servePut, nothing waits
on the waitvar, and I could not find a good way to make anything wait on
it.
2024-07-22 10:27:44 -04:00
Joey Hess
eb4fb388bd
only base64 non-utf8 2024-07-11 15:47:16 -04:00
Joey Hess
b290a72025
update deps 2024-07-11 14:51:45 -04:00
Joey Hess
97a2d0e4fb
use worker pool in withLocalP2PConnections
This allows multiple clients to be handled at the same time.
2024-07-11 14:37:52 -04:00
Joey Hess
68227154fb
switch HTTP P2P protocol to base64url
Base64 can include '/', and with UUIDs and keys both used in routes,
the encoding needs to avoid that. Use base64url everywhere in the HTTP
protocol for consistency.
2024-07-11 12:31:41 -04:00
Joey Hess
fc90270ba0
roughed in servePut 2024-07-11 12:20:07 -04:00
Joey Hess
1cff4c9f5b
oops, 2 datalength headers for put route 2024-07-11 12:12:33 -04:00
Joey Hess
3a55f0eec3
use DataLength 2024-07-11 11:57:55 -04:00
Joey Hess
14e0f778b7
simplify 2024-07-11 11:50:44 -04:00
Joey Hess
2228d56db3
serveGet invalidation 2024-07-11 11:42:32 -04:00
Joey Hess
80d2ffc79a
implement generic get server 2024-07-11 11:26:03 -04:00
Joey Hess
a7383b5c59
move serveruuid into routes
In particular the generic get route needs it, so that when a single http
server is serving multiple repositories, it knows what repository to
use.
2024-07-11 11:19:20 -04:00
Joey Hess
74c6175795
fix serveGet early handle close
Needed that waitv after all..
2024-07-11 09:55:17 -04:00
Joey Hess
2c13e6c165
fix annexworker shutdown on early client disconnect 2024-07-11 09:15:52 -04:00
Joey Hess
6e3d35744d
cleanup 2024-07-11 07:56:55 -04:00
Joey Hess
3b37b9e53f
fix serveGet hang
This came down to SendBytes waiting on the waitv. Nothing ever filled
it.

Only Annex.Proxy needs the waitv, and it handles filling it. So make it
optional.
2024-07-11 07:46:52 -04:00
Joey Hess
80fb5445b5
a little progress on serveGet hang
Now it gets to the validity checker, but it seems it never runs it.
2024-07-10 17:48:48 -04:00
Joey Hess
8cb1332407
update 2024-07-10 16:10:08 -04:00
Joey Hess
1e0f92a5a1
implemented serveGet and clientGet
Both are only at bare proof of concept stage. Still need to deal with
signaling validity and invalidity, and checking it.

And there's a bad bug: After -JN*2 requests, another request hangs!

So, I think it's failing to free up the Annex worker and end of request
lifetime.

Perhaps I need to use this:

https://docs.servant.dev/en/stable/cookbook/managed-resource/ManagedResource.html
2024-07-10 16:06:39 -04:00
Joey Hess
f9b7ce7224
add Annex worker pool to P2PHttp
This will be needed for get and store, since those need to run Annex
actions.

withLocalP2PConnections will also probably use it.
2024-07-10 12:19:47 -04:00
Joey Hess
d4b9aea87b
implement gettimestamp 2024-07-10 10:23:10 -04:00