Test::Harness features

[Ovid]

----- Original Message ----

> From: Steve Purkis <steve at purkis.ca>
>
> Start with extending TAP::Harness.  From what I can tell from your use  
> cases, my conversations with David Wheeler, and my own experience, we  
> have two main use cases to cater for in a plugin system:
> 
> 1. Custom formatting (Archive, HTML, JSON, File, etc.)
> 2. Custom sources (Archive, db, File, Nested TAP?, PHP, etc.)

Looking at this and thinking about core requirements, what we're really talking about is:

1.  Custom input
2.  Custom output

TAP:: might be a rather poor namespace because if someone wants to read POSIX test results, they can't.  And we can't even *dream* of a mixture of those.  But let's ignore the namespace for just a moment.  Given rather confused state of the testing world in regards to standardizing output, we have a unique opportunity to:

  TAP::Parser::Grammar::TAP
  TAP::Parser::Grammar::Ant
  TAP::Parser::Grammar::POSIX
  TAP::Parser::Grammar::TestNG
  ... and so on

Use cases:

1.  Test::Builder 2.0 will have (iirc) support for outputting POSIX test results.
2.  Hudson and other build systems typically use ant test results (as far as I understand, jUnit doesn't really have any output format.  The "jUnit XML" is really generated by ant. I might be rather mistaken).

Currently, we'd only need support for the TAP grammar, but supporting the other one's would give testers a field day.  And once parsing is normalized, custom output becomes the killer feature.  I could see many larger testing systems switching to this.

The problem would be a test suite which has combinations of POSIX, TAP, etc.


The custom "sources" originally mentioned would actually just be ways of streaming appropriate test results into the grammar engine.

And we still need test histories to round this out.  My App::Prove::History on github (they're down right now, so I can't link) is almost done, but it's very hackish internally because App::Prove really didn't support it too well and because I didn't understand the problem space well enough when I wrote it.  If you want a commit bit, drop me a line.  I'll happily give you one :)
 
Cheers,
Ovid
--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6