check-attr resource pool
Limited to min of -JN or number of CPU cores, because it will often be CPU bound, once it's read the gitignore file for a directory. In some situations it's more disk bound, but in any case it's unlikely to be the main bottleneck that -J is used to avoid. Eg, when dropping, this is used for numcopies checks, but the main bottleneck will be accessing the remotes to verify presence. So the user might decide to -J32 that, but having 32 check-attr processes would just waste however many filehandles they open, and probably worsen their performance due to CPU contention. Note that, I first tried just letting up to the -JN be started. However, even when it's no bottleneck at all, that still results in all of them being started. Why? Well, all the worker threads start up nearly simulantaneously, so there's a thundering herd..
This commit is contained in:
parent
cee6b344b4
commit
45fb7af21c
5 changed files with 47 additions and 16 deletions
|
@ -8,7 +8,7 @@
|
|||
{-# LANGUAGE BangPatterns #-}
|
||||
|
||||
module Utility.ResourcePool (
|
||||
ResourcePool,
|
||||
ResourcePool(..),
|
||||
mkResourcePool,
|
||||
mkResourcePoolNonConcurrent,
|
||||
withResourcePool,
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue