git-annex/debian
Joey Hess 7a33803193 Avoid pipeline stall when running git annex drop or fsck on a lot of files.
When it's stalled, there are 3 processes:

git annex
  git ls-files
  git check-attr

git-annex stalls trying to write to git check-attr, which stalls trying to
write to stdout (read by git-annex).

git ls-files does not seem to be involved directly; I've seen the stall when
it was still streaming out the file list, and after it had exited and
zombified.

The read and write are supposed to be handled by two different threads,
which pipeBoth forks off, thus avoiding deadlock. But it does deadlock.
(Certian signals unblock the deadlock for a while, then it stalls again.)

So, this is another case of WTF is the ghc IO manager doing today?
I avoid the issue by converting the writer to a separate process.

Possibly this was caused by some change in ghc 7 -- I'm offline and cannot
verify now, but I'm sure I used to be able to run git annex drop w/o it
hanging! And the code does not seem to have changed, except for commit
c1dc407941, which I tried reverting without
success. In fact, I reverted all the way back to 0.20110316 and still
saw the stall.

Update: Minimal test case:

import System.Cmd.Utils

main = do
	as <- checkAttr "blah" $ map show [1..100000]
	sequence $ map (putStrLn . show) as

checkAttr attr files = do
	(_, s) <- pipeBoth "git" params $ unlines files
	return $ lines s
	where
		params = ["check-attr", attr, "--stdin"]

Bug filed on ghc in debian, #624389
2011-04-27 23:18:35 -04:00
..
changelog Avoid pipeline stall when running git annex drop or fsck on a lot of files. 2011-04-27 23:18:35 -04:00
compat debianization 2010-10-16 16:41:24 -04:00
control Revert "Use haskell Crypto library instead of haskell SHA library.a" 2011-04-26 11:24:23 -04:00
copyright add license text 2011-03-22 21:23:35 -04:00
doc-base Add doc-base file. Closes: #621408 2011-04-06 21:57:22 -04:00
manpages update 2010-10-19 16:17:29 -04:00
NEWS No longer auto-upgrade to repository format 2, to avoid accidental upgrades, etc. Use git-annex upgrade when you're ready to run this version. 2011-03-19 18:33:39 -04:00
rules update 2010-10-27 15:08:46 -04:00