parallelize git-annex test for 25% speedup
Note the very weird workaround for what appears to be some kind of tasty bug, which causes a segfault. This is not new to this modification, I was seeing a segfault before at least intermittently when limiting git-annex test -p to only run a single test group. Also, the path from one test repo to a remote test repo used to be "../../foo", which somehow broke when moving the test repos from .t to .t/N. I don't actually quite understand how it used to work, but "../foo" seems correct and works in the new situation. Test output from the concurrent processes is not yet serialized. Should be easy to do using concurrent-output. More test groups will probably make the speedup larger. It would probably be best to have a larger number of test groups and divvy them amoung subprocesses numbered based on the number of CPU cores, perhaps times 2 or 3. Sponsored-by: Dartmouth College's Datalad project
This commit is contained in:
parent
62e6c6afb9
commit
8d14ce8f38
3 changed files with 132 additions and 80 deletions
|
@ -0,0 +1,17 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 8"""
|
||||
date="2022-03-14T18:52:09Z"
|
||||
content="""
|
||||
I have implemented parallelism as described in comment 7.
|
||||
|
||||
Currently there are 5 child processes, and the test runtime
|
||||
dropped from 444 to 334 seconds on my laptop.
|
||||
Splitting up the test groups further, so there are more child
|
||||
processes will probably improve that more.
|
||||
Remains to be seen if it helps on NFS much..
|
||||
|
||||
The `git-annex test` output is currently a mess, it needs to be serialized.
|
||||
Ran out of time to do that today, but the speed improvement is worth
|
||||
temporarily ugly output.
|
||||
"""]]
|
Loading…
Add table
Add a link
Reference in a new issue