check-attr resource pool
Limited to min of -JN or number of CPU cores, because it will often be CPU bound, once it's read the gitignore file for a directory. In some situations it's more disk bound, but in any case it's unlikely to be the main bottleneck that -J is used to avoid. Eg, when dropping, this is used for numcopies checks, but the main bottleneck will be accessing the remotes to verify presence. So the user might decide to -J32 that, but having 32 check-attr processes would just waste however many filehandles they open, and probably worsen their performance due to CPU contention. Note that, I first tried just letting up to the -JN be started. However, even when it's no bottleneck at all, that still results in all of them being started. Why? Well, all the worker threads start up nearly simulantaneously, so there's a thundering herd..
This commit is contained in:
parent
cee6b344b4
commit
45fb7af21c
5 changed files with 47 additions and 16 deletions
|
@ -14,6 +14,7 @@ import Annex.Action
|
|||
import Types.Concurrency
|
||||
import Types.WorkerPool
|
||||
import Types.CatFileHandles
|
||||
import Annex.CheckAttr
|
||||
import Remote.List
|
||||
|
||||
import Control.Concurrent
|
||||
|
@ -29,9 +30,11 @@ setConcurrency c = do
|
|||
cfh' <- case cfh of
|
||||
CatFileHandlesNonConcurrent _ -> liftIO catFileHandlesPool
|
||||
CatFileHandlesPool _ -> pure cfh
|
||||
cah <- mkConcurrentCheckAttrHandle c
|
||||
Annex.changeState $ \s -> s
|
||||
{ Annex.concurrency = c
|
||||
, Annex.catfilehandles = cfh'
|
||||
, Annex.checkattrhandle = Just cah
|
||||
}
|
||||
|
||||
{- Allows forking off a thread that uses a copy of the current AnnexState
|
||||
|
@ -67,7 +70,7 @@ dupState = do
|
|||
|
||||
st <- Annex.getState id
|
||||
-- Make sure that concurrency is enabled, if it was not already,
|
||||
-- so the resource pools are set up.
|
||||
-- so the concurrency-safe resource pools are set up.
|
||||
st' <- case Annex.concurrency st of
|
||||
NonConcurrent -> do
|
||||
setConcurrency (Concurrent 1)
|
||||
|
@ -77,7 +80,6 @@ dupState = do
|
|||
-- each thread has its own repoqueue
|
||||
{ Annex.repoqueue = Nothing
|
||||
-- avoid sharing open file handles
|
||||
, Annex.checkattrhandle = Nothing
|
||||
, Annex.checkignorehandle = Nothing
|
||||
}
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue