mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

cat-file resource pool

Browse source 
Avoid running a large number of git cat-file child processes when run with
a large -J value.

This implementation takes care to avoid adding any overhead to git-annex
when run without -J. When run with -J, there is a small bit of added
overhead, to manipulate the resource pool. That optimisation added a
fair bit of complexity.
This commit is contained in:
Joey Hess 2020-04-20 13:53:27 -04:00
parent 87b7b0f202
commit cee6b344b4
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
14 changed files with 243 additions and 47 deletions
Download patch file Download diff file Expand all files Collapse all files

2
doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__.mdwn
View file

@ -50,5 +50,3 @@ I will try to get a chance to troubleshoot it more to provide possibly more deta
[[!meta author=yoh]]
[[!tag projects/datalad]]
[[!tag moreinfo]]

15
doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__/comment_10_61622483d4f1962f191fb6a791c6817d._comment Normal file
View file

@ -0,0 +1,15 @@
[[!comment format=mdwn
username="joey"
subject="""comment 10"""
date="2020-04-20T17:24:53Z"
content="""
Implemented the cat-file pool. Capped at 2 cat-files of each distinct type,
so it will start a max of 8 no matter the -J level.
(Although cat-file can also be run in those repositories so there will be
more then.)
While testing, I noticed git-anenx drop -Jn starts n git check-attr
processes, so the same thing ought to be done with them. Leaving this bug open
for that, but I do think that the problem you reported should be fixed now.
"""]]