Test::Harness/Test::Builder features

[Ovid]

While we're waiting for Test::Builder 2.0 (due out shortly after Perl 6 ;), I am finding more and more that features we need at the BBC are features we're hacking in.  Specifically, we need nested TAP, but barring that, I've a hack in Test::Aggregate which fakes up some of the benefits.

With my latest development upload, you can get results like this (mocked up, obviously):

    #     ok - tests/pips-general.t (37 out of 336)
    #     ok - tests/creditrole-general.t (38 out of 336)
    #     ok - tests/format-general.t (39 out of 336)
    #     ok - tests/genre-general.t (40 out of 336)
    #     ok - tests/masterbrand-general.t (41 out of 336)
    #     ok - tests/promotion-general.t (42 out of 336)
    #     ok - tests/segment-general.t (43 out of 336)
    #     ok - tests/segmentevent-general.t (44 out of 336)
    #     ok - tests/service-general.t (45 out of 336)
    #     ok - tests/versiontype-general.t (46 out of 336)

So I'm effectively replacing the X/Y test count in the normal Test::Harness output.  However, the summary looks like this:

  All tests successful.
  Files=1, Tests=10947, 1198 wallclock secs ...
  Result: PASS

I'd really like to be able to override the report to have "Files=336".  However, I'm also teeing those results to a file and running this from a bash script:

    echo Failed aggregate tests:
    echo
    cat $results |awk '/not ok/ {print $5}'|grep '\.t$'|sed -e 's/^/    /'

That tells me which aggregated tests have failed, but it would be nice if there was a clean API to inject these results directly.  This API would no longer be necessary with nested TAP, but for the time being, I'm not seeing much of an option here.  Before I even think about moving forward, I'd love to hear some feedback about this.  I could potentially output nested TAP and have the Test::Harness handle it, or I could use new API features to inject results.  

Nested TAP would require more monkey-patching of Test::Builder (though I have commit rights, it doesn't mean that the patch would be accepted) and the latter would possibly involve code which gets thrown away with the advent of TB2.

Thoughts?

(Note:  my last aggregated test run took 16 minutes, but without aggregation takes over 30 to 40 minutes.  Algorithmic improvements offer only marginal improvement as the bulk of that extra time is from Catalyst, Moose, and DBIx::Class start up costs.)

Cheers,
Ovid
--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6