separate queue for cleanup actions

When running multiple concurrent actions, the cleanup phase is run in a
separate queue than the main action queue. This can make some commands
faster, because less time is spent on bookkeeping in between each file
transfer.

But as far as I can see, nothing will be sped up much by this yet, because
all the existing cleanup actions are very light-weight. This is just groundwork
for deferring checksum verification to cleanup time.

This change does mean that if the user expects -J2 will mean that they see no
more than 2 jobs running at a time, they may be surprised to see 4 in some
cases (if the cleanup actions are slow enough to notice).

It might also make sense to enable background cleanup without the -J,
for at least one cleanup action. Indeed, that's the behavior that -J1
has now. At some point in the future, it make make sense to make the
behavior with no -J the same as -J1. The only reason it's not currently
is that git-annex can build w/o concurrent-output, and also any bugs
in concurrent-output (such as perhaps misbehaving on non-VT100 compatible
terminals) are avoided by default by only using it when -J is used.
This commit is contained in:
Joey Hess 2019-06-05 17:54:35 -04:00
parent c04b2af3e1
commit 659640e224
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
6 changed files with 128 additions and 46 deletions

View file

@ -11,7 +11,6 @@ import Annex
import Annex.Common
import Annex.Action
import qualified Annex.Queue
import Types.WorkerPool
import qualified Data.Map as M
@ -43,9 +42,8 @@ dupState :: Annex AnnexState
dupState = do
st <- Annex.getState id
return $ st
{ Annex.workers = UnallocatedWorkerPool
-- each thread has its own repoqueue
, Annex.repoqueue = Nothing
{ Annex.repoqueue = Nothing
-- avoid sharing eg, open file handles
, Annex.catfilehandles = M.empty
, Annex.checkattrhandle = Nothing