fix git-annex test -p

test: When limiting tests to run with -p, work around tasty limitation by
automatically including dependent tests.

This fixes a reversion because it didn't used to use dependencies and
forced tasty to run the init tests first. That changed when parallelizing
the test suite.

It will sometimes do a little more work than strictly required,
because it adds init tests deps when limited to eg quickcheck tests,
which don't depend on them. But this only adds a few seconds work.

Sponsored-by: Dartmouth College's Datalad project

CHANGELOG

@@ -19,6 +19,8 @@ git-annex (10.20220505) UNRELEASED; urgency=medium
   * Improve an error message displayed in that situation.
   * Prevent git-annex init incorrectly reinitializing the repository in
     that situation.
+  * test: When limiting tests to run with -p, work around tasty limitation
+    by automatically including dependent tests.
 
  -- Joey Hess <id@joeyh.name>  Thu, 05 May 2022 15:08:07 -0400
 

Test.hs

@@ -252,7 +252,7 @@ testRemote testvariants remotetype setupremote =
 {- These tests set up the test environment, but also test some basic parts
  - of git-annex. They are always run before the repoTests. -}
 initTests :: TestTree
-initTests = testGroup "Init Tests"
+initTests = testGroup initTestsName
 	[ testCase "init" test_init
 	, testCase "add" test_add
 	]
@@ -339,7 +339,7 @@ repoTests note numparts = map mk $ sep
 	]
   where
 	mk l = testGroup groupname (initTests : map adddep l)
-	adddep = Test.Tasty.after AllSucceed (groupname ++ ".Init Tests")
+	adddep = Test.Tasty.after AllSucceed (groupname ++ "." ++ initTestsName)
 	groupname = "Repo Tests " ++ note
 	sep = sep' (replicate numparts [])
 	sep' (p:ps) (l:ls) = sep' (ps++[l:p]) ls

Test/Framework.hs

@@ -15,6 +15,7 @@ import Test.Tasty.HUnit
 import Test.Tasty.Options
 import Test.Tasty.Ingredients.Rerun
 import Test.Tasty.Ingredients.ConsoleReporter
+import qualified Test.Tasty.Patterns.Types as TP
 import Options.Applicative.Types
 import Control.Concurrent
 import Control.Concurrent.Async
@@ -724,10 +725,12 @@ parallelTestRunner' numjobs opts mkts
 	| otherwise = go =<< Utility.Env.getEnv subenv
   where
 	subenv = "GIT_ANNEX_TEST_SUBPROCESS"
+
 	-- Make more parts than there are jobs, because some parts
 	-- are larger, and this allows the smaller parts to be packed
 	-- in more efficiently, speeding up the test suite overall.
 	numparts = numjobs * 2
+
 	worker rs nvar a = do
 		(n, m) <- atomically $ do
 			(n, m) <- readTVar nvar
@@ -738,6 +741,7 @@ parallelTestRunner' numjobs opts mkts
 			else do
 				r <- a n
 				worker (r:rs) nvar a
+	
 	go Nothing = withConcurrentOutput $ do
 		ensuredir tmpdir
 		crippledfilesystem <- fst <$> Annex.Init.probeCrippledFileSystem'
@@ -753,7 +757,7 @@ parallelTestRunner' numjobs opts mkts
 		args <- getArgs
 		pp <- Annex.Path.programPath
 		termcolor <- hSupportsANSIColor stdout
-		let ps = if useColor (lookupOption (tastyOptionSet opts)) termcolor
+		let ps = if useColor (lookupOption tastyopts) termcolor
 			then "--color=always":args
 			else "--color=never":args
 		let runone n = do
@@ -783,16 +787,29 @@ parallelTestRunner' numjobs opts mkts
 		Just (n, crippledfilesystem, adjustedbranchok) -> setTestEnv $ do
 			let ts = mkts numparts crippledfilesystem adjustedbranchok opts
 			let t = topLevelTestGroup [ ts !! (n - 1) ]
-			case tryIngredients ingredients (tastyOptionSet opts) t of
+			case tryIngredients ingredients tastyopts t of
 				Nothing -> error "No tests found!?"
 				Just act -> ifM act
 					( exitSuccess
 					, exitFailure
 					)
 	
+	tastyopts = case lookupOption (tastyOptionSet opts) of
+		-- Work around limitation of tasty; when tests to run
+		-- are limited to a pattern, it does not include their
+		-- dependencies. So, add another pattern including the
+		-- init tests, which are a dependency of most tests.
+		TestPattern (Just p) -> 
+			setOption (TestPattern (Just (TP.Or p (TP.ERE initTestsName))))
+				(tastyOptionSet opts)
+		TestPattern Nothing -> tastyOptionSet opts
+
 topLevelTestGroup :: [TestTree] -> TestTree
 topLevelTestGroup = testGroup "Tests"
 
+initTestsName :: String
+initTestsName = "Init Tests"
+
 tastyParser :: [TestTree] -> ([String], Parser Test.Tasty.Options.OptionSet)
 #if MIN_VERSION_tasty(1,3,0)
 tastyParser ts = go

doc/bugs/tasty_test_limiting_broken_by_concurrency.mdwn

@@ -1,14 +1,19 @@
 The changes to make `git-annex test` concurrent have
 broken using eg `git-annex test -p 'concurrent get of dup key regression'`
 
-The problem is that tasty is being run with a subset of tests in each
+It looks like the repo setup test is not being run, even though it's
-runner, so most of them don't know about the test they're being limited to
-perform.
-
-There either needs to be a way to disable concurrency (eg, run all tests
-in one runner with -J1), or the code detect when tasty is limited, and
-automatically disable concurrency.
-
-Also, it looks like the repo setup test is not being run, even though it's
 supposed to be a dependency of the test it was limited to.
 --[[Joey]]
+
+> Oh, that seems to be a limitation of tasty. From its docs:
+	
+	If Test B depends on Test A, remember that either of them may be 
+	filtered out using the --pattern option. Collecting the dependency
+	info happens after filtering. Therefore, if Test A is filtered out,
+	Test B will run unconditionally, and if Test B is filtered out,
+	it simply won't run.
+
+> This works: `git-annex test -p '/concurrent get of dup key regression/ || /Init Tests/'`
+> 
+> Ok, I was able to work around this by having git-annex test add the latter
+> pattern automatically. [[done]] --[[Joey]]