improve git command queue flushing with time limit
So that eg, addurl of several large files that take time to download will update the index for each file, rather than deferring the index updates to the end. In cases like an add of many smallish files, where a new file is being added every few seconds. In that case, the queue will still build up a lot of changes which are flushed at once, for best performance. Since the default queue size is 10240, often it only gets flushed once at the end, same as before. (Notice that updateQueue updated _lastchanged when adding a new item to the queue without flushing it; that is necessary to avoid it flushing the queue every 5 minutes in this case.) But, when it takes more than a 5 minutes to add a file, the overhead of updating the index immediately is probably small, so do it after each file. This avoids git-annex potentially taking a very very long time indeed to stage newly added files, which can be annoying to the user who would like to get on with doing something with the files it's already added, eg using git mv to rename them to a better name. This is only likely to cause a problem if it takes say, 30 seconds to update the index; doing an extra 30 seconds of work after every 5 minute file add would be less optimal. Normally, updating the index takes significantly less time than that. On a SSD with 100k files it takes less than 1 second, and the index write time is bound by disk read and write so is not too much worse on a hard drive. So I hope this will not impact users, although if it does turn out to, the time limit could be made configurable. A perhaps better way to do it would be to have a background worker thread that wakes up every 60 seconds or so and flushes the queue. That is made somewhat difficult because the queue can contain Annex actions and so this would add a new source of concurrency issues. So I'm trying to avoid that approach if possible. Sponsored-by: Erik Bjäreholt on Patreon
This commit is contained in:
parent
fe31951e5e
commit
c2e46f4707
3 changed files with 46 additions and 13 deletions
|
@ -78,7 +78,8 @@ get = maybe new return =<< getState repoqueue
|
|||
|
||||
new :: Annex (Git.Queue.Queue Annex)
|
||||
new = do
|
||||
q <- Git.Queue.new . annexQueueSize <$> getGitConfig
|
||||
sz <- annexQueueSize <$> getGitConfig
|
||||
q <- liftIO $ Git.Queue.new sz Nothing
|
||||
store q
|
||||
return q
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue