Commit graph

44415 commits

Author SHA1 Message Date
Joey Hess
703a70cafa
avoid watchFileSize running backward
This is groundwork for using watchFileSize for downloads from external
special remotes.

In Annex.Content.downloadUrl, this potentially avoids jitter in the
progress meter. When downloading with conduit, the meter gets updated based
on both the size of the file, and on the data flowing through conduit.
If that has not yet been flushed to the file, it seems possible for the
meter to run backwards when meter is updated with the file size.
It's probably only a few kb of jitter, so may not be visible.

Sponsored-by: Dartmouth College's DANDI project
2024-01-19 14:11:27 -04:00
jstritch
4276cb2015 Added a comment 2024-01-19 17:44:39 +00:00
Joey Hess
856c28508d
Merge branch 'master' of ssh://git-annex.branchable.com 2024-01-19 13:01:05 -04:00
Joey Hess
df35f70801
tweak stall detection scaling
Refactored to allow offline experimentation, and ended up changing the
allowedvariation (aka fudge factor) to 3. 10 seems too high, and 1.5 too low.

Scale earlier, so even if the first chunk takes less than the configured
time period, allowance is made that later chunks might transfer slower.
Decided to use the same allowedvariation to decide when to start
scaling.

Smoothed the scaling out.

Some examples:

ghci> upscale (BwRate 10 (Duration 60)) 25
BwRate 13 (Duration {durationSeconds = 75})
-- A small scaling upwards after 1/3rd the time. Not noticable.
ghci> upscale (BwRate 10 (Duration 60)) 60
BwRate 30 (Duration {durationSeconds = 180})
-- At the configured time, 3x scaling.
ghci> upscale (BwRate 10 (Duration 60)) 120
BwRate 60 (Duration {durationSeconds = 360})
-- A typical upscaling, here a 1 minute duration became 6 minutes
-- due to the first chunk taking 2 minutes to transfer.
ghci> upscale (BwRate 10 (Duration 60)) 600
BwRate 300 (Duration {durationSeconds = 1800})
-- Here the first chunk took 10 minutes to transfer, so it will
-- take 30 minutes to detect a stall.

Sponsored-by: Dartmouth College's DANDI project
2024-01-19 12:58:41 -04:00
imlew
0270f4d8ed Added a comment 2024-01-18 23:09:21 +00:00
yarikoptic
3e047f05cf Added a comment 2024-01-18 21:32:30 +00:00
Joey Hess
e61af28acf
comment 2024-01-18 17:18:19 -04:00
Joey Hess
3ef766c444
comment 2024-01-18 17:16:18 -04:00
Joey Hess
0721fe3463
Merge branch 'master' of ssh://git-annex.branchable.com 2024-01-18 17:13:05 -04:00
Joey Hess
c2634e7df2
automatically adjust stall detection period
Improve annex.stalldetection to handle remotes that update progress less
frequently than the configured time period.

In particular, this makes remotes that don't report progress but are
chunked work when transferring a single chunk takes longer than the
specified time period.

Any remotes that just have very low update granulatity would also be
handled by this.

The change to Remote.Helper.Chunked avoids an extra progress update when
resuming an interrupted upload. In that case, the code saw first Nothing
and then Just the already transferred number of bytes, which defeated this
new heuristic. This change will mean that, when resuming an interrupted
upload to a chunked remote that does not do its own progress reporting, the
progress display does not start out displaying the amount sent so far,
until after the first chunk is sent. This behavior change does not seem
like a major problem.

About the scalefudgefactor, it seems reasonable to expect subsequent chunks
to take no more than 1.5 times as long as the first chunk to transfer.
Could set it to 1, but then any chunk taking a little longer would be
treated as a stall. 2 also seems a likely value. Even 10 might be fine?

Sponsored-by: Dartmouth College's DANDI project
2024-01-18 17:12:10 -04:00
yarikoptic
4fe3e59525 Added a comment 2024-01-18 19:00:51 +00:00
Joey Hess
8f655f7953
improve annex.stalldetection documentation 2024-01-18 14:38:16 -04:00
imlew
cd986d613b 2024-01-18 18:25:48 +00:00
Joey Hess
931920c426
comment 2024-01-18 13:14:17 -04:00
Joey Hess
3ca1e036ed
open todo 2024-01-18 13:11:28 -04:00
Joey Hess
ae4177ec4e
comment 2024-01-18 13:01:04 -04:00
Joey Hess
52d4f36b2b
document that --listen specifies an IP address
Avoid users getting confused and thinking this allows specifying the
port.

Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
2024-01-18 12:58:00 -04:00
Joey Hess
adb8b320e3
cleanup 2024-01-18 12:42:26 -04:00
Joey Hess
e765d3e24c
import: --message/-m option 2024-01-18 12:41:44 -04:00
Joey Hess
162c772703
comment 2024-01-18 12:16:48 -04:00
Joey Hess
05026e2053
not a bug 2024-01-18 12:11:10 -04:00
Joey Hess
ccc64bc8ff
fixed 2024-01-18 11:56:03 -04:00
Joey Hess
fa7f598284
fix windows build 2024-01-18 11:54:59 -04:00
Joey Hess
517904b770
Merge branch 'master' of ssh://git-annex.branchable.com 2024-01-18 11:53:20 -04:00
imlew
38e79ba1d7 2024-01-17 14:32:12 +00:00
nobodyinperson
7cf0d88fa1 Mention annextimelog 2024-01-16 23:54:27 +00:00
jkniiv
92bb192c3a Added a comment 2024-01-16 23:23:27 +00:00
jkniiv
b9339d9f5d Added a comment: I can confirm 2024-01-16 22:28:22 +00:00
Joey Hess
285073712a
close 2024-01-16 14:38:49 -04:00
Joey Hess
f6cf2dec4c
disk free checking for unsized keys
Improve disk free space checking when transferring unsized keys to
local git remotes. Since the size of the object file is known, can
check that instead.

Getting unsized keys from local git remotes does not check the actual
object size. It would be harder to handle that direction because the size
check is run locally, before anything involving the remote is done. So it
doesn't know the size of the file on the remote.

Also, transferring unsized keys to other remotes, including ssh remotes and
p2p remotes don't do disk size checking for unsized keys. This would need a
change in protocol.

(It does seem like it would be possible to implement the same thing for
directory special remotes though.)

In some sense, it might be better to not ever do disk free checking for
unsized keys, than to do it only sometimes. A user might notice this
direction working and consider it a bug that the other direction does not.
On the other hand, disk reserve checking is not implemented for most
special remotes at all, and yet it is implemented for a few, which is also
inconsistent, but best effort. And so doing this best effort seems to make
some sense. Fundamentally, if the user wants the size to always be checked,
they should not use unsized keys.

Sponsored-by: Brock Spratlen on Patreon
2024-01-16 14:29:10 -04:00
Joey Hess
94dcbdf0ac
Merge branch 'master' of ssh://git-annex.branchable.com 2024-01-16 11:56:16 -04:00
Joey Hess
c15fa17635
optimise adjustTree when adding many TreeItems (take 2)
The old code traversed the list of addtreeitems once per subdirectory in
the tree, so could get quite slow. Converting to Map lookups sped it up
significantly.

In my test case, git-annex import used to take about 2 minutes, when
calling adjustTree to add back excluded files to the imported tree. This
dropped it down to 6 seconds. Of which 4 seconds are the actual
enumeration of the contents of the remote, so really only 2 seconds for
this.

The path prefix map is a bit suboptimal memory-wise, since items get
stored in the map once per subdirectory on the path to the item. It
would perhaps be better to use a tree data structure.

Also it's suboptimal memory-wise that it builds two maps, as well
as retaining a reference to addtreeitems. I could not see a way around
that though.

This is a fixed version of commit 2c86651180.
It fixes a test suite reversion.

Sponsored-by: Jack Hill on Patreon
2024-01-16 11:53:57 -04:00
kdm9
c3989e3e05 typo fix (renote -> remote) 2024-01-16 14:36:56 +00:00
yarikoptic
8fd6c3f1b1 windows build is broken 2024-01-15 15:38:43 +00:00
TTTTAAAx
795b3f3d45 Added a comment 2024-01-15 01:43:28 +00:00
aurelien@f0d0a0c7da69eff6badf0464898f0a859f69114d
34cf5be43d Added a comment 2024-01-14 23:18:43 +00:00
jaharmi
986f8f6be4 rename forum/Initial_macOS_setup_between_two_Macs.mdwn to forum/Initial_macOS_setup_between_two_local_Macs.mdwn 2024-01-14 20:28:54 +00:00
jaharmi
d80e4e42f9 2024-01-14 20:28:24 +00:00
jaharmi
6d499d51c4 2024-01-14 20:27:44 +00:00
jstritch
acca5ab06b 2024-01-14 16:23:13 +00:00
jstritch
629f570288 2024-01-14 16:15:36 +00:00
Joey Hess
3fbc5d9c7e
Merge branch 'master' of ssh://git-annex.branchable.com 2024-01-12 13:52:04 -04:00
Joey Hess
dda4cb372c
update 2024-01-12 13:51:59 -04:00
Joey Hess
7e69063a29
support annex.shared-sop-command for encryption=shared
This works well, and it interoperates with gpg in my testing (although some
SOP commands might choose to use a profile that does not so caveat emptor).

Note that for creating the Cipher, gpg --gen-random is still used. SOP
does not have an eqivilant, and as long as the user has gpg around,
which seems likely, it doesn't matter that it uses gpg here, it's not being
used for encryption. That seemed better than implementing a second way
to get high quality entropy, at least for now.

The need for the sop command to run in an empty directory has each call
to encrypt and decrypt creating a new temporary directory. That is some
unncessary overhead, though probably swamped by the overhead of running
the sop command. This could be improved in the future by passing an
already empty directory to them, or a sufficiently empty directory
(.git/annex/tmp would probably suffice).

Sponsored-by: Brett Eisenberg on Patreon
2024-01-12 13:31:18 -04:00
Joey Hess
dd3e779020
more groundwork for StatelessOpenPGP
no behavior changes
2024-01-12 13:11:36 -04:00
nobodyinperson
e8a1ad5253 Added a comment: Try datalad 2024-01-12 09:23:16 +00:00
jstritch
181e14467f 2024-01-11 18:24:52 +00:00
Joey Hess
2fb200a110
clarify that combined hook program is deprecated, not entire hook special remote 2024-01-11 11:04:07 -04:00
TTTTAAAx
9a83b2c64f Added a comment 2024-01-11 14:15:58 +00:00
TTTTAAAx
d6cca10efd Added a comment 2024-01-11 13:58:27 +00:00