mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

avoid potentially very long bwlimit delay at start

Browse source 
I first saw this getting with -J2 over ssh, but later saw it also
without the -J2. It was resuming, and the calulated unboundDelay was
many minutes. The first update of the meter jumped to some large value,
because of the resuming, and so it thought the BW was super fast.

Avoid by waiting until the second meter update.

Might be a good idea to also guard for the delay being many seconds
and avoid waiting. But how many? If BW is legitimately super fast, and a
remote happens to read more than a 32kb or so chunk at a time, it could
in theory download megabytes or gigabytes of data before the first meter
update. It would actually be appropriate then to delay for a long time,
if the desired BW was low. Could make up some numbers that are sane now,
but tech may improve.

(BTW, pleased to see bwlimit does work with -J. I had worried that
it might not, if the meter update happened in a different thread than
the downloading, but it's done in the same thread.)

Sponsored-by: Brett Eisenberg on Patreon
This commit is contained in:
Joey Hess 2021-09-22 18:38:15 -04:00
parent 94a9116175
commit 64cac1a721
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 17 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

22
Utility/Metered.hs
View file

@ -397,23 +397,27 @@ bwLimitMeterUpdate bwlimit duration meterupdate
| bwlimit <= 0 = return meterupdate
| otherwise = do
nowtime <- getPOSIXTime
mv <- newMVar (nowtime, 0)
mv <- newMVar (nowtime, Nothing)
return (mu mv)
where
mu mv n@(BytesProcessed i) = do
endtime <- getPOSIXTime
(starttime, previ) <- takeMVar mv
(starttime, mprevi) <- takeMVar mv
case mprevi of
Just previ -> do
let runtime = endtime - starttime
let currbw = fromIntegral (i - previ) / runtime
let pausescale = if currbw > bwlimit'
then (currbw / bwlimit') - 1
else 0
unboundDelay (floor (runtime * pausescale * msecs))
Nothing -> return ()
let runtime = endtime - starttime
let currbw = fromIntegral (i - previ) / runtime
let pausescale = if currbw > bwlimit'
then (currbw / bwlimit') - 1
else 0
unboundDelay (floor (runtime * pausescale * msecs))
meterupdate n
nowtime <- getPOSIXTime
putMVar mv (nowtime, i)
putMVar mv (nowtime, Just i)
bwlimit' = fromIntegral (bwlimit * durationSeconds duration)
msecs = fromIntegral oneSecond

11
doc/todo/bwlimit.mdwn
View file

@ -10,9 +10,9 @@ works, it will probably work to put the delay in there. --[[Joey]]
[[confirmed]]
> Implemented and works well.
>
> A local git remote, when resuming an interrupted
> Implemented and works well. [[done]] --[[Joey]]
> Note: A local git remote, when resuming an interrupted
> transfer, has to hash the file (with default annex.verify settings),
> and that hashing updates the progress bar, and so the bwlimit can kick
> in and slow down that initial hashing, before any data copying begins.
@ -20,8 +20,7 @@ works, it will probably work to put the delay in there. --[[Joey]]
> remote you're wanting to limit disk IO. Only reason it might not be ok
> is if the intent is to limit IO to the disk containing the remote
> but not the one containing the annex repo. (This also probably
> holds for the directory special remote.)
>
> holds for the directory special remote.)
> Other remotes, including git over ssh, when resuming don't have that
> problem. Looks like chunked special remotes narrowly avoid it, just
> because their implementation choose to not do incremental verification
@ -37,5 +36,3 @@ works, it will probably work to put the delay in there. --[[Joey]]
> tend to be 32kb or so, and the pauses a small fraction of a second. So
> mentioning this only for completeness.) --[[Joey]]
[[done]]