mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Fix a lock file descriptor leak that could occur when running commands like git-annex add with -J

Browse source 
Bug was introduced as part of a different FD leak fix in version 6.20160318.
This commit is contained in:
Joey Hess 2020-07-21 15:30:47 -04:00
parent fd8339005a
commit ac56a5c2a0
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
5 changed files with 54 additions and 27 deletions
Download patch file Download diff file Expand all files Collapse all files

2
doc/bugs/Adding_files_fails_with_--jobs_more_than_2.mdwn
View file

@ -62,3 +62,5 @@ failed
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I've been having quite a bit of fun revisiting git-annex and datalad after quite a while, and I'm really excited to see how far things have come. I'm pretty close to adopting these tools in my data science group after a pretty exhaustive search of related technologies like quilt, dvc, and attempts to role my own solution. Using Github + the S3 special remote w/versioning enabled is just about the best solution to dataset tracking I've come across.
> [[fixed|done]] --[[Joey]]

25
doc/bugs/Adding_files_fails_with_--jobs_more_than_2/comment_2_45702c5a87e891a0dde5459b5156c57e._comment Normal file
View file

@ -0,0 +1,25 @@
[[!comment format=mdwn
username="joey"
subject="""analysis"""
date="2020-07-21T18:35:08Z"
content="""
Utility.LockPool.STM.releaseLock seems to be where the problem is.
It waits to close a lock FD if some other thread is using the lock.
But, this means that, if enough threads are run that the lock is
always in use by one thread, it will never close it. Meanwhile, each
lockShared call opens the lock file anew, accumilating another FD.
3334130368829ad2041006560e578f1f876f68e4 is at fault, indeed.
That commit mentions that it would be better to have two calls to
lockShared only open the file once, but that it would be difficult to do
that atomically. Perhaps there is a way to do that... It didn't seem easy
to do this time either.
Alternatively, registerCloseLockFile currently takes an action that closes
one lock FD, and just combines that to its existing close action with `>>`.
So it builds up one big action that closes all the FDs. Instead, make each
lock handle contain its close action, and have releaseLock only release the
one it was called with. Implemented this, and it solved it.
"""]]