avoid splitting repo tests into too small parts around -J16
The initTests have to be run once per part, and a point of diminishing returns can be reached where more work is being done to set up for 1 or 2 tests than to run them. This is better than a hard cap of -J8 or so, because it lets other things than these particular tests still be parallelized at -J16. Sponsored-by: Dartmouth College's Datalad project
This commit is contained in:
parent
5e6cb47bd8
commit
4f6c6114fb
2 changed files with 37 additions and 1 deletions
|
@ -0,0 +1,28 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2022-11-07T17:07:52Z"
|
||||
content="""
|
||||
I wondered if running the add and init tests once per worker was causing
|
||||
a diminishing returns at higher -J levels. Since at -J16, each worker runs
|
||||
the add and init tests followed by only 2 or 3 other tests, the time spent
|
||||
on the add and init tests becomes more and more significant.
|
||||
|
||||
Timings from my laptop (with 4 cores):
|
||||
|
||||
-J16 250 seconds
|
||||
-J8 212 seconds
|
||||
-J4 214 seconds
|
||||
-J2 307 seconds
|
||||
|
||||
So a small diminishing returns at -J16.
|
||||
|
||||
After some improvements:
|
||||
|
||||
-J16 223 seconds
|
||||
-J8 218 seconds
|
||||
-J4 214 seconds
|
||||
|
||||
That won't solve smaug being so overloaded though.
|
||||
If anything, it will just make it get to some other test before it times out..
|
||||
"""]]
|
Loading…
Add table
Add a link
Reference in a new issue