[658] Generate batches of tests with different pathologies and run different prove invocations against them .

[Andy Armstrong]

On 2 Oct 2007, at 02:09, andy at hexten.net wrote:
> Generate batches of tests with different pathologies and run  
> different prove invocations against them.

And my results (MacBook Pro, 2.2Ghz Core Duo):

Running prove -rQ  pt/fast
All tests successful.
Files=100, Tests=336000, 31 wallclock secs (28.21 usr  2.14 sys +   
0.28 cusr  0.43 csys = 31.06 CPU)
Result: PASS
Running prove -rQ -j9 pt/fast
All tests successful.
Files=100, Tests=336000, 41 wallclock secs (37.68 usr  2.71 sys +   
0.28 cusr  0.44 csys = 41.11 CPU)
Result: PASS
Running prove -rQ -j9 --fork pt/fast
All tests successful.
Files=100, Tests=336000, 17 wallclock secs ( 0.28 usr  0.09 sys +  
29.02 cusr  2.67 csys = 32.06 CPU)
Result: PASS
Running prove -rQ -j4 --fork pt/fast
All tests successful.
Files=100, Tests=336000, 17 wallclock secs ( 0.28 usr  0.08 sys +  
28.53 cusr  2.56 csys = 31.45 CPU)
Result: PASS
----------------------------------------
Running prove -rQ  pt/fickle
All tests successful.
Files=2500, Tests=2500, 30 wallclock secs ( 6.08 usr 15.07 sys +   
3.85 cusr 10.35 csys = 35.35 CPU)
Result: PASS
Running prove -rQ -j9 pt/fickle
All tests successful.
Files=2500, Tests=2500, 27 wallclock secs ( 6.42 usr 14.59 sys +   
3.78 cusr 10.55 csys = 35.34 CPU)
Result: PASS
Running prove -rQ -j9 --fork pt/fickle
All tests successful.
Files=2500, Tests=2500, 17 wallclock secs ( 0.61 usr  0.30 sys +  
10.13 cusr 19.39 csys = 30.43 CPU)
Result: PASS
Running prove -rQ -j4 --fork pt/fickle
All tests successful.
Files=2500, Tests=2500, 17 wallclock secs ( 0.63 usr  0.26 sys +  
10.12 cusr 18.98 csys = 29.99 CPU)
Result: PASS
----------------------------------------
Running prove -rQ  pt/greedy
All tests successful.
Files=30, Tests=30, 31 wallclock secs ( 0.07 usr  0.09 sys + 29.82  
cusr  0.30 csys = 30.28 CPU)
Result: PASS
Running prove -rQ -j9 pt/greedy
All tests successful.
Files=30, Tests=30, 16 wallclock secs ( 0.07 usr  0.08 sys + 30.02  
cusr  0.30 csys = 30.47 CPU)
Result: PASS
Running prove -rQ -j9 --fork pt/greedy
All tests successful.
Files=30, Tests=30, 16 wallclock secs ( 0.02 usr  0.02 sys + 30.18  
cusr  0.47 csys = 30.69 CPU)
Result: PASS
Running prove -rQ -j4 --fork pt/greedy
All tests successful.
Files=30, Tests=30, 17 wallclock secs ( 0.02 usr  0.01 sys + 30.14  
cusr  0.44 csys = 30.61 CPU)
Result: PASS
----------------------------------------
Running prove -rQ  pt/gross
All tests successful.
Files=1, Tests=336000, 30 wallclock secs (28.07 usr  1.64 sys +  0.13  
cusr  0.01 csys = 29.85 CPU)
Result: PASS
Running prove -rQ -j9 pt/gross
All tests successful.
Files=1, Tests=336000, 42 wallclock secs (38.97 usr  2.98 sys +  0.13  
cusr  0.01 csys = 42.09 CPU)
Result: PASS
Running prove -rQ -j9 --fork pt/gross
All tests successful.
Files=1, Tests=336000, 29 wallclock secs ( 0.25 usr  0.05 sys + 27.40  
cusr  1.68 csys = 29.38 CPU)
Result: PASS
Running prove -rQ -j4 --fork pt/gross
All tests successful.
Files=1, Tests=336000, 29 wallclock secs ( 0.25 usr  0.05 sys + 27.40  
cusr  1.70 csys = 29.40 CPU)
Result: PASS
----------------------------------------
Running prove -rQ  pt/lazy
All tests successful.
Files=30, Tests=30, 30 wallclock secs ( 0.10 usr  0.13 sys +  0.05  
cusr  0.17 csys =  0.45 CPU)
Result: PASS
Running prove -rQ -j9 pt/lazy
All tests successful.
Files=30, Tests=30,  4 wallclock secs ( 0.10 usr  0.12 sys +  0.05  
cusr  0.16 csys =  0.43 CPU)
Result: PASS
Running prove -rQ -j9 --fork pt/lazy
All tests successful.
Files=30, Tests=30,  7 wallclock secs ( 0.02 usr  0.02 sys +  0.20  
cusr  0.37 csys =  0.61 CPU)
Result: PASS
Running prove -rQ -j4 --fork pt/lazy
All tests successful.
Files=30, Tests=30,  9 wallclock secs ( 0.02 usr  0.02 sys +  0.17  
cusr  0.34 csys =  0.55 CPU)
Result: PASS
----------------------------------------
Running prove -rQ  pt
All tests successful.
Files=2661, Tests=674560, 160 wallclock secs (63.39 usr 26.40 sys +  
34.19 cusr 11.64 csys = 135.62 CPU)
Result: PASS
Running prove -rQ -j9 pt
All tests successful.
Files=2661, Tests=674560, 141 wallclock secs (88.21 usr 31.91 sys +  
34.38 cusr 12.71 csys = 167.21 CPU)
Result: PASS
Running prove -rQ -j9 --fork pt
All tests successful.
Files=2661, Tests=674560, 127 wallclock secs ( 1.16 usr  0.38 sys +  
99.50 cusr 26.41 csys = 127.45 CPU)
Result: PASS
Running prove -rQ -j4 --fork pt
All tests successful.
Files=2661, Tests=674560, 114 wallclock secs ( 1.18 usr  0.38 sys +  
97.67 cusr 26.40 csys = 125.63 CPU)
Result: PASS
----------------------------------------

Interesting that pt/lazy is slower with --fork. I suspect that the  
forking code isn't distributing the tests as evenly among the  
available workers as the multiplexing code. That can probably be fixed.

-- 
Andy Armstrong, Hexten