Avoid pipeline stall when running git annex drop or fsck on a lot of files.

When it's stalled, there are 3 processes:

git annex
  git ls-files
  git check-attr

git-annex stalls trying to write to git check-attr, which stalls trying to
write to stdout (read by git-annex).

git ls-files does not seem to be involved directly; I've seen the stall when
it was still streaming out the file list, and after it had exited and
zombified.

The read and write are supposed to be handled by two different threads,
which pipeBoth forks off, thus avoiding deadlock. But it does deadlock.
(Certian signals unblock the deadlock for a while, then it stalls again.)

So, this is another case of WTF is the ghc IO manager doing today?
I avoid the issue by converting the writer to a separate process.

Possibly this was caused by some change in ghc 7 -- I'm offline and cannot
verify now, but I'm sure I used to be able to run git annex drop w/o it
hanging! And the code does not seem to have changed, except for commit
c1dc407941, which I tried reverting without
success. In fact, I reverted all the way back to 0.20110316 and still
saw the stall.

Update: Minimal test case:

import System.Cmd.Utils

main = do
	as <- checkAttr "blah" $ map show [1..100000]
	sequence $ map (putStrLn . show) as

checkAttr attr files = do
	(_, s) <- pipeBoth "git" params $ unlines files
	return $ lines s
	where
		params = ["check-attr", attr, "--stdin"]

Bug filed on ghc in debian, #624389
This commit is contained in:
Joey Hess 2011-04-27 22:10:57 -04:00
parent 39966ba4ee
commit 7a33803193
2 changed files with 11 additions and 1 deletions

View file

@ -78,6 +78,7 @@ import Data.Word (Word8)
import Codec.Binary.UTF8.String (encode)
import Text.Printf
import Data.List (isInfixOf, isPrefixOf)
import System.Exit
import Utility
@ -482,7 +483,14 @@ checkAttr repo attr files = do
-- in its output back to relative.
cwd <- getCurrentDirectory
let absfiles = map (absPathFrom cwd) files
(_, s) <- pipeBoth "git" (toCommand params) $ join "\0" absfiles
(_, fromh, toh) <- hPipeBoth "git" (toCommand params)
_ <- forkProcess $ do
hClose fromh
hPutStr toh $ join "\0" absfiles
hClose toh
exitSuccess
hClose toh
s <- hGetContents fromh
return $ map (topair $ cwd++"/") $ lines s
where
params = gitCommandLine repo [Param "check-attr", Param attr, Params "-z --stdin"]

2
debian/changelog vendored
View file

@ -4,6 +4,8 @@ git-annex (0.20110426) UNRELEASED; urgency=low
Debian stable.
* Added rsync special remotes. This could be used, for example, to
store annexed content on rsync.net (encrypted naturally). Or anywhere else.
* Bugfix: Avoid pipeline stall when running git annex drop or fsck on a
lot of files. Possibly only occured with ghc 7.
-- Joey Hess <joeyh@debian.org> Tue, 26 Apr 2011 11:23:54 -0400