Test::Harness 2.99_02 vs. Parrot

[Andy Dougherty]

Just to see what would happen, I tried installing Test::Harness 2.99_02
(using INSTALLDIRS='perl' to overwrite the old Test::Harness 2.56).
I then tried 'make test' for parrot.  There are a lot of differences.
Whether or not these are parrot's fault, I don't know, but I thought it might
be worthwhile to point them out.

There are trivial differences in the number of '.'s included in output lines.
After I eliminate those, here's what I see:

	$ diff TH-2.56.log TH-2.99_03.log | wc
	712

Of those, 43 entries are of the form (here in diff -u format):

--- TH-2.56.log	Mon Sep 10 12:35:34 2007
+++ TH-2.99_03.log	Mon Sep 10 11:05:23 2007

 t/op/cc_state.......................................ok
-t/op/cmp-nonbranch..................................ok
+t/op/cmp-nonbranch..................................Option -I not known
+parrot -[abcCEfgGhjprStvVwy.] [-d [FLAGS]] [-D [FLAGS]][-O [level]] [-o FILE] <file>
+ Dubious, test returned 1 (wstat 256, 0x100)
+ No subtests run 
 t/op/comp...........................................ok
 t/op/debuginfo......................................ok

What's happening is that some of the tests (e.g. t/op/cmp-nonbranch.t)
start with
	#!./parrot
but the test harness is trying to run parrot with the 
non-existent -I option.

Approximately 54 of the differences are of this form:

--- TH-2.56.log	Mon Sep 10 12:35:34 2007
+++ TH-2.99_03.log	Mon Sep 10 11:05:23 2007

-t/postconfigure/05-trace............................skipped
-        all skipped: Tests irrelevant unless configuration completed with tracing requested
+t/postconfigure/05-trace............................ok


 t/compilers/imcc/imcpasm/opt1.......................ok
-        1/78 skipped: constant concat N/Y
 t/compilers/imcc/imcpasm/opt2.......................ok
-        1/6 skipped: loop opt disabled for now

where various bits of skip information are ommitted.  Sometimes, that
skip information is actually useful to know:

-t/op/64bit..........................................skipped
-        all skipped: 64bit INTVAL platforms only
+t/op/64bit..........................................ok

Lastly, the "Summary" at the end is vastly expanded, to the point where
it's unclear if it's useful as a "summary" anymore.  To be specific,
the version 2.56 summary is 8 lines long, fits comfortably on one
screen, and gives a useful summary of the tests.

Failed Test                    Stat Wstat Total Fail  Failed  List of Failed
-------------------------------------------------------------------------------
t/compilers/pge/p5regex/p5rx.t    0     3   960  806  83.96%  558-960
t/pmc/complex.t                   5  1280    53    5   9.43%  32 42-43 46-47
t/src/hash.t                      1   256    11    1   9.09%  6
t/src/list.t                      1   256     2    1  50.00%  1
 (7 subtests UNEXPECTEDLY SUCCEEDED), 11 tests and 456 subtests skipped.
Failed 4/341 test scripts, 98.83% okay. 410/7824 subtests failed, 94.76% okay.

By contrast, the "Summary" from version 2.99_02 is 230 lines long.
It does indeed contain more information, and much of it is useful, but
at 230 lines, it's not as useful as a summary.  Here are some excerpts:

Test Summary Report
-------------------
t/configure/004-configure.t                     (Wstat: 0 Tests: 30 Failed: 0)
  Tests skipped: 28-29
t/configure/017-revision_no_DEVELOPING.t        (Wstat: 0 Tests: 13 Failed: 0)
  Tests skipped: 4-12
t/configure/018-revision.t                      (Wstat: 0 Tests: 7 Failed: 0)
  Tests skipped: 4-6

[ ... lots more ... ]

t/stm/queue.t                                   (Wstat: 0 Tests: 4 Failed: 0)
  Tests skipped: 4
t/codingstd/copyright.t                         (Wstat: 0 Tests: 2 Failed: 0)
  Tests skipped: 2
Files=341, Tests=4821, 2477 wallclock secs (1443.79 cusr + 589.12 csys = 2032.91 CPU)
Result: FAIL
Failed 48/341 test programs. 7/4821 subtests failed.


As I said, I don't know how much of this is parrot's fault for abusing
the current Test::Harness, but before setting INSTALLDIRS='perl' on a
release, nor how much of it is deliberate design decisions.  I just
thought you ought to at least be aware of it.

(Please CC me on any replies, as I am not subscribed to this list, and
attempts to subscribe via http://www.hexten.net/mailman/listinfo/tapx-dev
failed, and I lazily gave up at that point.)

-- 
    Andy Dougherty		doughera at lafayette.edu