CI to automate build of cargo lockfiles on different Alpine releases for git-annex aports https://gitlab.alpinelinux.org/alpine/aports/-/tree/master/community/git-annex
Find a file
Joey Hess ed0afbc36b
avoid concurrent threads trying to take pid lock at same time
Seem there are several races that happen when 2 threads run PidLock.tryLock
at the same time. One involves checkSaneLock of the side lock file, which may
be deleted by another process that is dropping the lock, causing checkSaneLock
to fail. And even with the deletion disabled, it can still fail, Probably due
to linkToLock failing when a second thread overwrites the lock file.

The same can happen when 2 processes do, but then one process just fails
to take the lock, which is fine. But with 2 threads, some actions where failing
even though the process as a whole had the pid lock held.

Utility.LockPool.PidLock already maintains a STM lock, and since it uses
LockShared, 2 threads can hold the pidlock at the same time, and when
the first thread drops the lock, it will remain held by the second
thread, and so the pid lock file should not get deleted until the last
thread to hold it drops the lock. Which is the right behavior, and why a
LockShared STM lock is used in the first place.

The problem is that each time it takes the STM lock, it then also calls
PidLock.tryLock. So that was getting called repeatedly and concurrently.

Fixed by noticing when the shared lock is already held, and stop calling
PidLock.tryLock again, just use the pid lock that already exists then.

Also, LockFile.PidLock.tryLock was deleting the pid lock when it failed
to take the lock, which was entirely wrong. It should only drop the side
lock.

Sponsored-by: Dartmouth College's Datalad project
2021-12-01 17:14:39 -04:00
Annex remove unused import 2021-11-23 16:15:57 -04:00
Assistant rename to avoid warning 2021-10-18 16:25:00 -04:00
Backend convert Key to ShortByteString 2021-10-05 20:20:08 -04:00
Build test borg special remote 2021-09-03 13:18:07 -04:00
CmdLine add git-annex filter-process 2021-11-04 15:02:36 -04:00
Command export: Avoid unncessarily re-exporting non-annexed files that were already exported 2021-11-29 14:02:38 -04:00
Config add filter.annex.process to bypassSmudgeConfig 2021-11-04 15:23:07 -04:00
Database avoid using a second db connection for writes 2021-10-20 12:32:46 -04:00
debian Enable building with git-annex benchmark by default 2020-10-02 13:57:10 -04:00
doc avoid concurrent threads trying to take pid lock at same time 2021-12-01 17:14:39 -04:00
Git support git 2.34.0's handling of merge conflict between annexed and non-annexed file 2021-11-22 16:10:24 -04:00
Limit fix longstanding indeterminite preferred content for duplicated file problem 2021-05-24 14:07:05 -04:00
Logs fix test suite failure when run with LANG=C 2021-08-18 17:36:00 -04:00
Messages tighter --json parsing for metadata 2021-11-01 14:42:37 -04:00
P2P factor out IncrementalHasher from IncrementalVerifier 2021-11-09 12:33:22 -04:00
Remote git-lfs gitlab interoperability fix 2021-11-10 13:51:11 -04:00
RemoteDaemon remove support code for ancient git-annex-shell version 2021-10-11 16:07:08 -04:00
standalone remove osx stack.yaml 2021-10-18 13:15:47 -04:00
static Revert "remove newlines from static js and css" 2014-06-13 02:20:39 -04:00
templates cache the serialization of a Key 2019-11-22 17:49:16 -04:00
Test add test case for readonly remote 2021-08-30 16:39:02 -04:00
Types have setConcurrency stop any running git coprocesses 2021-11-19 12:00:39 -04:00
Upgrade convert Key to ShortByteString 2021-10-05 20:20:08 -04:00
Utility avoid concurrent threads trying to take pid lock at same time 2021-12-01 17:14:39 -04:00
.ghci avoid ghci warning on tabs 2019-01-21 12:35:44 -04:00
.gitattributes update changelog location 2016-08-22 23:54:11 -04:00
.gitignore stack.yaml: Updated to lts-16.10 2020-08-24 14:11:37 -04:00
.mailmap alias for Yaroslav @ yahoo to mailmap 2015-09-09 12:38:59 -04:00
Annex.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Assistant.hs assistant: Fix a crash on startup by avoiding using forkProcess 2021-05-12 15:08:03 -04:00
Backend.hs remove unused imports 2021-10-05 21:27:06 -04:00
bash-completion.bash bash completion fix 2018-11-12 13:23:05 -04:00
Benchmark.hs --size-limit exit 101 2021-06-04 16:43:47 -04:00
BuildFlags.hs Removed the S3 and WebDAV build flags 2020-09-08 12:42:59 -04:00
BuildInfo.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
CHANGELOG avoid concurrent threads trying to take pid lock at same time 2021-12-01 17:14:39 -04:00
CmdLine.hs --size-limit exit 101 2021-06-04 16:43:47 -04:00
Command.hs addon commands 2021-02-02 16:32:49 -04:00
Common.hs finish fixing removeLink on windows 2020-11-24 13:20:44 -04:00
Config.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
COPYRIGHT update year 2021-01-04 12:45:19 -04:00
Creds.hs avoid getting creds from environment during autoenable 2021-03-17 09:41:12 -04:00
Crypto.hs convert Key to ShortByteString 2021-10-05 20:20:08 -04:00
git-annex.cabal releasing package git-annex version 8.20211123 2021-11-23 15:20:24 -04:00
git-annex.hs include git-annex-shell back in 2019-12-02 11:51:52 -04:00
git-union-merge.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Git.hs reinject: Fix crash when reinjecting a file from outside the repository 2021-10-01 14:06:34 -04:00
Key.hs convert Key to ShortByteString 2021-10-05 20:20:08 -04:00
Limit.hs New matching options --excludesamecontent and --includesamecontent 2021-05-25 13:08:18 -04:00
Logs.hs filter-branch working aside from some edge cases 2021-05-17 13:24:58 -04:00
Makefile remove TAGS target 2021-07-02 14:38:28 -04:00
Messages.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
NEWS fix dist in latest NEWS entry 2020-09-14 12:05:12 -04:00
README use https 2014-11-06 14:20:10 -04:00
Remote.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Setup.hs Added git-remote-tor-annex, which allows git pull and push to the tor hidden service. 2016-11-21 17:27:38 -04:00
stack.yaml update git-lfs version to match git-annex.cabal 2021-11-16 12:54:29 -04:00
Test.hs fix test failure with git version 2.34.0 2021-11-22 13:30:03 -04:00
Types.hs plumb VerifyConfig into retrieveKeyFile 2021-08-17 12:43:13 -04:00
Upgrade.hs propagate git-annex -c on to transferrer child process 2020-12-15 11:36:25 -04:00

git-annex allows managing files with git, without checking the file
contents into git. While that may seem paradoxical, it is useful when
dealing with files larger than git can currently easily handle, whether due
to limitations in memory, checksumming time, or disk space.

For documentation, see doc/ or <https://git-annex.branchable.com/>