diff --git a/doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__/comment_9_5db907d77c5f5490c1bdc8e51387a9e9._comment b/doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__/comment_9_5db907d77c5f5490c1bdc8e51387a9e9._comment new file mode 100644 index 0000000000..851c37da81 --- /dev/null +++ b/doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__/comment_9_5db907d77c5f5490c1bdc8e51387a9e9._comment @@ -0,0 +1,27 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 9""" + date="2020-04-20T15:54:15Z" + content=""" +Thinking about this over the weekend, I had two ideas: + +* The worker pool has an AnnexState for each thread. If those could be + partitioned so eg perfom stage is always run by the same threads, + then when only one stage needs cat-file, the overall number of cat-file + processes would be reduced by 1/3rd. + + This might be the least resource intensive approach. But, as threads + transition between stages, their AnnexState necessarily does too, + and the cleanup stage might need some state change made in the perform + stage, so swapping out the perform AnnexState for a cleanup one + seems hard to accomplish. + +* Could have a pool of cat-files, and just have worker threads block until + one is available. This would let it be pinned to the -J number, or + event to a smaller number. + + Seems likely that only 2 or 3 in the cat-file pool will + maximise concurrency, because it's not a major bottleneck most of the + time, and when it is the actual bottleneck is probably disk IO and so + won't be helped by more (and likely more only increase unncessary seeks). +"""]]