When bup split fails, display its stderr

It seems worth noting here that I emailed bup's author about bup split
being noisy on stderr even with -q in approximately 2011. That never got
fixed. Its current repo on github only accepts pull requests, not bug
reports. Needing to add such complexity to deal with such a longstanding
unfixed issue is not fun.

Sponsored-by: Kevin Mueller on Patreon

CHANGELOG

@@ -14,6 +14,7 @@ git-annex (10.20220725) UNRELEASED; urgency=medium
   * Work around bug in git 2.37 that causes a segfault
     when when core.untrackedCache is set, and broke git-annex init.
   * Improve output when storing to bup.
+  * When bup split fails, display its stderr.
 
  -- Joey Hess <id@joeyh.name>  Mon, 25 Jul 2022 15:35:45 -0400
 

Remote/Bup.hs

@@ -1,6 +1,6 @@
 {- Using bup as a remote.
  -
- - Copyright 2011-2020 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2022 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -13,6 +13,7 @@ import qualified Data.Map as M
 import qualified Data.ByteString as S
 import qualified Data.ByteString.Lazy as L
 import Data.ByteString.Lazy.UTF8 (fromString)
+import Control.Concurrent.Async
 
 import Annex.Common
 import qualified Annex
@@ -160,19 +161,26 @@ store r buprepo = byteStorer $ \k b p -> do
 		    cmd = (proc "bup" (toCommand params))
 			{ std_in = CreatePipe
 			, std_out = UseHandle nullh
-			-- bup split is noisy to stderr even with the -q
-			-- option.
-			, std_err = UseHandle nullh
+			, std_err = CreatePipe
 			}
 		    feeder = \h -> do
 			meteredWrite p (S.hPut h) b
 			hClose h
 		in withCreateProcess cmd (go feeder cmd)
   where
-	go feeder p (Just h) _ _ pid =
-		forceSuccessProcess p pid
-			`after`
-		feeder h
+	go feeder p (Just inh) _ (Just errh) pid = do
+		-- bup split is noisy to stderr even with the -q
+		-- option. But when bup fails, the stderr needs
+		-- to be displayed.
+		(feedresult, erroutput) <- tryNonAsync (feeder inh)
+			`concurrently` hGetContentsStrict errh
+		waitForProcess pid >>= \case
+			ExitSuccess -> case feedresult of
+				Right () -> return ()
+				Left e -> throwM e
+			ExitFailure n -> giveup $ 
+				showCmd p ++ " exited " ++ show n ++
+					" (stderr output: " ++ erroutput ++ ")"
 	go _ _ _ _ _ _ = error "internal"
 
 retrieve :: BupRepo -> Retriever

Utility/Process.hs

@@ -31,6 +31,7 @@ module Utility.Process (
 	stdoutHandle,
 	stderrHandle,
 	processHandle,
+	showCmd,
 	devNull,
 ) where
 

doc/bugs/bup_often_errors_out_when_-J___62___1/comment_1_1731cfb2662890299ca6daeacdd4bb47._comment

@@ -0,0 +1,22 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-08-05T15:57:26Z"
+ content="""
+This seems most likely to be a concurrency issue in bup itself, or in how
+it uses git.
+
+However, I have not been able to reproduce it so far. I tried making 8
+different files each 100 mb, and was able to copy them all to a bup repository
+concurrently with no failures.
+
+That was with bup 0.32. What is your bup version?
+
+It would be good to know what error message bup outputs.
+Unfortunately, since `bup split -q` is noisy on stderr, git-annex
+sinks it to /dev/null and so you don't see it.
+
+I've changed git-annex to display the stderr when `bup split` fails.
+So if you can get an updated build of git-annex, and reproduce this problem,
+we'll be able to see if this is a bug in bup, or what.
+"""]]