avoid splitting repo tests into too small parts around -J16

The initTests have to be run once per part, and a point of diminishing
returns can be reached where more work is being done to set up for 1 or
2 tests than to run them.

This is better than a hard cap of -J8 or so, because it lets other
things than these particular tests still be parallelized at -J16.

Sponsored-by: Dartmouth College's Datalad project
This commit is contained in:
Joey Hess 2022-11-07 14:44:51 -04:00
parent 5e6cb47bd8
commit 4f6c6114fb
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 37 additions and 1 deletions

View file

@ -0,0 +1,28 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2022-11-07T17:07:52Z"
content="""
I wondered if running the add and init tests once per worker was causing
a diminishing returns at higher -J levels. Since at -J16, each worker runs
the add and init tests followed by only 2 or 3 other tests, the time spent
on the add and init tests becomes more and more significant.
Timings from my laptop (with 4 cores):
-J16 250 seconds
-J8 212 seconds
-J4 214 seconds
-J2 307 seconds
So a small diminishing returns at -J16.
After some improvements:
-J16 223 seconds
-J8 218 seconds
-J4 214 seconds
That won't solve smaug being so overloaded though.
If anything, it will just make it get to some other test before it times out..
"""]]