check-attr resource pool

Limited to min of -JN or number of CPU cores, because it will often be
CPU bound, once it's read the gitignore file for a directory.

In some situations it's more disk bound, but in any case it's unlikely
to be the main bottleneck that -J is used to avoid. Eg, when dropping,
this is used for numcopies checks, but the main bottleneck will be
accessing the remotes to verify presence. So the user might decide to
-J32 that, but having 32 check-attr processes would just waste however
many filehandles they open, and probably worsen their performance due to
CPU contention.

Note that, I first tried just letting up to the -JN be started. However,
even when it's no bottleneck at all, that still results in all of them
being started. Why? Well, all the worker threads start up nearly
simulantaneously, so there's a thundering herd..
This commit is contained in:
Joey Hess 2020-04-21 10:38:44 -04:00
parent cee6b344b4
commit 45fb7af21c
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
5 changed files with 47 additions and 16 deletions

View file

@ -14,6 +14,7 @@ import Annex.Action
import Types.Concurrency
import Types.WorkerPool
import Types.CatFileHandles
import Annex.CheckAttr
import Remote.List
import Control.Concurrent
@ -29,9 +30,11 @@ setConcurrency c = do
cfh' <- case cfh of
CatFileHandlesNonConcurrent _ -> liftIO catFileHandlesPool
CatFileHandlesPool _ -> pure cfh
cah <- mkConcurrentCheckAttrHandle c
Annex.changeState $ \s -> s
{ Annex.concurrency = c
, Annex.catfilehandles = cfh'
, Annex.checkattrhandle = Just cah
}
{- Allows forking off a thread that uses a copy of the current AnnexState
@ -67,7 +70,7 @@ dupState = do
st <- Annex.getState id
-- Make sure that concurrency is enabled, if it was not already,
-- so the resource pools are set up.
-- so the concurrency-safe resource pools are set up.
st' <- case Annex.concurrency st of
NonConcurrent -> do
setConcurrency (Concurrent 1)
@ -77,7 +80,6 @@ dupState = do
-- each thread has its own repoqueue
{ Annex.repoqueue = Nothing
-- avoid sharing open file handles
, Annex.checkattrhandle = Nothing
, Annex.checkignorehandle = Nothing
}