Commit graph

2233 commits

Author SHA1 Message Date
Joey Hess
126daf949d
DATA-PRESENT working for exporttree=yes remotes
Since the annex-tracking-branch is pushed first, git-annex has already
updated the export database when the DATA-PRESENT arrives. Which means
that just using checkPresent is enough to verify that there is some file
on the special remote in the export location for the key.

So, the simplest possible implementation of this happened to work!

(I also tested it with chunked specialremotes, which also works, as long
as the chunk size used is the same as the configured chunk size. In that
case, the lack of a chunk log is not a problem. Doubtful this will ever
make sense to use with a chunked special remote though, that gets pretty
deep into re-implementing git-annex.)

Updated the client side upload tip with a missing step, and reorged for clarity.
2024-10-30 13:55:47 -04:00
Joey Hess
ccbc5189b5
Fix hang when receiving a large file into a proxied special remote
Only indicate that we're done with the bytestring once it all gets written.
Otherwise, the end of it may get garbage collected before we can process
it, leading to a hang.

This seems to have been introduced in commit
cdc4bd7443. Which oddly was trying to fix a
very similar problem, but specific to a cluster node. In that commit,
things got out of order, with it signaling it was done with the bytestring
before it has written all of it to the file.

My test case for this bug is a directory special remote
with a file being sent to it via a proxy accessed via ssh or http.
The file was 10 mb, and it hung on the last few kb of it not being
received.

I've also tested this fix in the case of proxying to a cluster node
directory special remote over http, which was the case
cdc4bd7443 was dealing with.
2024-10-30 12:29:37 -04:00
Joey Hess
f19ebabe89
support DATA-PRESENT when proxtying for special remotes
There is a TODO left in the code for exporttree special remotes. If
possible, it should check if one of the export locations contains the
content of the key.
2024-10-29 14:55:31 -04:00
Joey Hess
8baccda98f
Merge branch 'master' into streamproxy 2024-10-22 09:49:28 -04:00
Joey Hess
2c14181bcb
better name for LinkPresentAdjustment 2024-10-21 15:42:01 -04:00
Joey Hess
d9b4bf4224
added retrieveKeyFileInOrder and ORDERED to external special remote protocol
I anticipate lots of external special remote programs will neglect
implementing this. Still, it's the right thing to do to assume that some
of them may write files out of order. Probably most external special
remotes will not be used with a proxy. When someone is using one with a
proxy, they can always get it fixed to send ORDERED.
2024-10-15 15:40:14 -04:00
Joey Hess
f920d90781
smaller delay in proxy streamer
A one second delay made it seem really choppy and slow when the special
remote was sending content fairly steadily but was bottlenecked on
running gpg on 10 mb chunks.

This does not appreciably increase CPU, although of course if the
special remote is very slow it will add up over time.

It would perhaps be better to use inotify, like tailVerify does.
2024-10-15 14:45:19 -04:00
Joey Hess
54fcc2ec51
fix logic error 2024-10-15 14:28:47 -04:00
Joey Hess
edaed18e4c
Sped up proxied downloads from special remotes, by streaming
Currently works for special remotes that don't use fileRetriever. Ones that
do will download to another filename and rename it into place, defeating
the streaming.

This actually benchmarks slightly slower when getting a large file from
a fast proxied special remote. However, when the proxied special remote
is slow, it will be a big win.
2024-10-15 12:25:15 -04:00
Joey Hess
10216b44d2
use NonEmpty for dirHashes
This avoids 4 uses of head.
2024-09-26 18:15:00 -04:00
Joey Hess
783e910d0c
sim: Add metadata command
Only really needed for completeness, preferred content expressions can
match against metadata.
2024-09-26 12:20:37 -04:00
Joey Hess
6a95e4edad
sim: support "--" as comment
Using this in my sim files that are also mdwn files to avoid comments
being displayed as headers.
2024-09-25 14:47:32 -04:00
Joey Hess
8e94b75a61
support simulating clusters
Without actually simulating cluster implementation at all. Instead, only
the essential fact that cluster gateways know what changes they have
made to each node of a cluster. That is enough for sims like
sizebalanced_cluster.
2024-09-25 14:06:41 -04:00
Joey Hess
b9214d4162
Revert "sim: add commands for cluster management"
This reverts commit 344141da63.

Rethinking this
2024-09-25 12:11:03 -04:00
Joey Hess
344141da63
sim: add commands for cluster management
Clusters are not actually simulated yet.
2024-09-25 11:48:22 -04:00
Joey Hess
8047128591
sim: quiesce before freezing or ending
Probably a good idea for freezing, but especially I hope this fixes a
problem with git-annex sim run that caused it to sometimes crash in
removeDirectoryRecursive with directory not empty, presumably because a
thread was writing there at the same time.
2024-09-24 16:46:09 -04:00
Joey Hess
540bd5e1ab
sim: added run subcommand
And a nice sim of random preferred content expressions.
2024-09-24 12:06:34 -04:00
Joey Hess
9571162057
sim: add stepstable 2024-09-24 11:50:24 -04:00
Joey Hess
4ed58d7894
sim: random preferred content expression generation 2024-09-24 11:23:23 -04:00
Joey Hess
ee3d6502bb
prevent action or step from simulating running on a special remote
Without any connections, the step command will not try to do any actions
on a special remote.

But even without any connections, it's still possible for a drop action
explicitly run "on" the special remote to do, when numcopies = 0 or
there is a trusted repo. So guard all actions against running on a
special remote too.
2024-09-24 10:15:56 -04:00
Joey Hess
7cc4312695
fix state overwrite bug
I have needed to excercise a lot of care in threading st through, and I
got it wrong here. Probably using a state monad would be a good idea.
2024-09-24 10:00:38 -04:00
Joey Hess
d3a3c722c9
oops 2024-09-23 16:02:39 -04:00
Joey Hess
eec07aec68
sim: avoid step looking for new actions every time
Once it has a list of actions, it can perform them all.

A disappointing optimisation at least in my test case, which it sped up
by less than 1 second out of 12. But still it did make it faster.
2024-09-23 15:50:47 -04:00
Joey Hess
969e6c2747
sped up sim step by about 200%
Noticed that it was quite slow compared with things like action
sendwanted. Guessed that the slowdown is largely due to every step
doing a simulated git pull/push.

So, rather than always doing a pull/push, only do those when no actions
are found without doing a pull/push.

This does mean that step will sometimes experience a split brain
situation, but that seems like a good thing? Because step ought to
explore as many possible scenarios as it reasonably can.
2024-09-23 15:45:47 -04:00
Joey Hess
6b040ed32d
allow complex shell commands 2024-09-23 15:04:33 -04:00
Joey Hess
7bc8c2bfeb
sim visit as first-class command
Allows using it in a sim file.
2024-09-23 13:09:35 -04:00
Joey Hess
6cf9a101b8
sim: Fix size tracking for balanced preferred content 2024-09-23 12:42:32 -04:00
Joey Hess
e9d4cef10f
sim: fix state loss bug 2024-09-20 18:11:37 -04:00
Joey Hess
f5f7b4a936
avoid adding redundant present/notpresent to sim history 2024-09-20 15:45:05 -04:00
Joey Hess
e9c59eceb8
bugfixes
sim stabilization works now
2024-09-20 15:39:52 -04:00
Joey Hess
19b966f0fd
sim: better step
On each step, find all the actions that could be done, and pick one of them
to do.

Should detect stability, but that is broken.
2024-09-20 15:23:34 -04:00
Joey Hess
31679e3e9f
set simRootDirectory on restore
It's a relative directory and the cwd may be different. Or the repo
could have been moved.
2024-09-20 15:11:55 -04:00
Joey Hess
bab330de33
remove sim log file 2024-09-20 15:03:54 -04:00
Joey Hess
f061ae92fb
sim: implement addtree 2024-09-20 10:34:52 -04:00
Joey Hess
6751f23978
sim: fix get bug
When getting from a remote, have to check that the repo doing the
getting thinks the remote contains the key, but also that the remote
actually does. Before this bug fix, it would get from a repo that used
to have the key, but that had dropped it since the last git pull.
2024-09-17 14:29:49 -04:00
Joey Hess
e568ac96b7
record initial seed in sim log
Unless the log starts with a command that records a seed.
2024-09-17 13:49:50 -04:00
Joey Hess
02f0996e25
git-annex sim log 2024-09-17 13:43:11 -04:00
Joey Hess
b85965cb3c
sim: implement dropunwantedfrom 2024-09-17 13:35:35 -04:00
Joey Hess
eb5fad4e79
fix ActionDropUnwanted
Now tested working
2024-09-17 11:55:57 -04:00
Joey Hess
e0128b6362
sim: add missing parser for action sendwanted 2024-09-17 11:40:40 -04:00
Joey Hess
ae19ad0fa1
sim: fix changing settings bug 2024-09-17 11:38:26 -04:00
Joey Hess
4c7db31c20
addmulti 2024-09-17 11:22:14 -04:00
Joey Hess
2a16796a1c
move pull/push/sync into getSimActionComponents
As well as being a more pleasing implementation than I managed
yesterday, this allows for those actions to be run concurrently in the
sim.
2024-09-17 10:54:44 -04:00
Joey Hess
7d27a8ea1a
sim concurrency 2024-09-17 10:37:22 -04:00
Joey Hess
92c10045d1
implement ActionPull/Pull/Sync
Somewhat unsatisfying implementation, but all the alternatives were
worse.
2024-09-16 16:58:03 -04:00
Joey Hess
c420ec9364
sim: add action repo sync command 2024-09-16 16:48:21 -04:00
Joey Hess
52891711d2
git-annex sim command is working
Had to add Read instances to Key and NumCopies and some other similar
types. I only expect to use those in serializing a sim. Of course, this
risks that implementation changes break reading old data. For a sim,
that would not be a big problem.
2024-09-12 16:10:52 -04:00
Joey Hess
7e8274c6b7
implemented ActionDropUnwanted
Not tested yet. This emulates the same checking that is done when
dropping. Note that when dropping from a special remote it is not able
to make a locked copy.
2024-09-12 10:44:31 -04:00
Joey Hess
68e52f6ec0
convert simRepoState to use UUID as key 2024-09-12 10:29:32 -04:00
Joey Hess
7b931df475
fully preserve input format of connect and disconnect commands
Just lifted the sim file as a DSL into the type level for that.
2024-09-11 21:01:13 -04:00