parallelize git-annex test for 25% speedup

Note the very weird workaround for what appears to be some kind of tasty
bug, which causes a segfault. This is not new to this modification,
I was seeing a segfault before at least intermittently when limiting
git-annex test -p to only run a single test group.

Also, the path from one test repo to a remote test repo used to be
"../../foo", which somehow broke when moving the test repos from .t to
.t/N. I don't actually quite understand how it used to work, but
"../foo" seems correct and works in the new situation.

Test output from the concurrent processes is not yet serialized.
Should be easy to do using concurrent-output.

More test groups will probably make the speedup larger. It would
probably be best to have a larger number of test groups and divvy them
amoung subprocesses numbered based on the number of CPU cores, perhaps
times 2 or 3.

Sponsored-by: Dartmouth College's Datalad project
This commit is contained in:
Joey Hess 2022-03-14 15:24:37 -04:00
parent 62e6c6afb9
commit 8d14ce8f38
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
3 changed files with 132 additions and 80 deletions

View file

@ -0,0 +1,17 @@
[[!comment format=mdwn
username="joey"
subject="""comment 8"""
date="2022-03-14T18:52:09Z"
content="""
I have implemented parallelism as described in comment 7.
Currently there are 5 child processes, and the test runtime
dropped from 444 to 334 seconds on my laptop.
Splitting up the test groups further, so there are more child
processes will probably improve that more.
Remains to be seen if it helps on NFS much..
The `git-annex test` output is currently a mess, it needs to be serialized.
Ran out of time to do that today, but the speed improvement is worth
temporarily ugly output.
"""]]