mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

use watchFileSize in Remote.External.retrieveKeyFile

Browse source 
external: Monitor file size when getting content from external special
remotes and use that to update the progress meter, in case the external
special remote program does not report progress.

This relies on 703a70cafa to prevent ever
running the meter backwards.

Sponsored-by: Dartmouth College's DANDI project
This commit is contained in:
Joey Hess 2024-01-19 14:31:54 -04:00
parent 8da85fd3a3
commit c02df79248
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
4 changed files with 47 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
CHANGELOG
View file

@ -15,6 +15,9 @@ git-annex (10.20231228) UNRELEASED; urgency=medium
* import: --message/-m option.
* Improve annex.stalldetection to handle remotes that update progress
less frequently than the configured time period.
* external: Monitor file size when getting content from external
special remotes and use that to update the progress meter,
in case the external special remote program does not report progress.
-- Joey Hess <id@joeyh.name> Fri, 29 Dec 2023 11:52:06 -0400

2
Remote/External.hs
View file

@ -235,7 +235,7 @@ storeKeyM external = fileStorer $ \k f p ->
retrieveKeyFileM :: External -> Retriever
retrieveKeyFileM external = fileRetriever $ \d k p ->
either giveup return =<< go d k p
either giveup return =<< watchFileSize d p (go d k)
where
go d k p = handleRequestKey external (\sk -> TRANSFER Download sk (fromRawFilePath d)) k (Just p) $ \resp ->
case resp of

22
doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_5_9fc2d7f4b39615e43bce3993e0a6e647._comment Normal file
View file

@ -0,0 +1,22 @@
[[!comment format=mdwn
username="joey"
subject="""comment 6"""
date="2024-01-19T17:31:41Z"
content="""
Re asymmetric connections, stall detection doesn't need to be
configured anywhere near the total available bandwidth, so generally it
would be ok to configure it for a slow uplink even when a fast downlink is
available. And if you're only interested in detecting total stalls
(as opposed to some kind of "too slow" condition), "1k" per some unit of
time is sufficient.
But maybe it does make sense to be able to configure it to abandon slow
downloads but allow slow uploads. For example, `git-annex get` with
the content on several remotes, where the download speed from one
remote is often fast but occasionally slows down, and another remote
is consistently medium speed.
So you might set "10gb/1m" for that remote, knowing that if it is slow
it will abort the download from it and fall back to the medium speed remote.
But when sending content *to* the variable speed remote, would not want to
give up only because it was a little slow.
"""]]

21
doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_6_1b2eb9993e220082f3be5dd568deef3e._comment Normal file
View file

@ -0,0 +1,21 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2024-01-19T17:01:27Z"
content="""
> "download", where git-annex actually IIRC keeps its monitoring of the file size as it is arriving
git-annex does do file size monitoring
in a few cases (when running `cp`, and when downloading
an url with `curl`), but it does not do it for downloads
in general, and not for external special remotes specifically.
If it did for external special remotes, and if rclone grows the
destination file as it downloads, rather than atomically at the end
(which I have not verified), that would have avoided this problem.
Probably it will avoids this type of problem for downloads
from the majority of external special remotes. Though of course not for
uploads.
Ok, implemented that for external special remotes.
"""]]