Test::Harness features

[Steve Purkis]

Sorry for the delay, it's been hectic again...

--
  Steve Purkis

On 10 Jun 2009, at 12:01, Ovid <publiustemp-tapx at yahoo.com> wrote:

>
> ----- Original Message ----
>
>> From: Steve Purkis <steve at purkis.ca>
>>
>> Start with extending TAP::Harness.  From what I can tell from your  
>> use
>> cases, my conversations with David Wheeler, and my own experience, we
>> have two main use cases to cater for in a plugin system:
>>
>> 1. Custom formatting (Archive, HTML, JSON, File, etc.)
>> 2. Custom sources (Archive, db, File, Nested TAP?, PHP, etc.)
>
> Looking at this and thinking about core requirements, what we're  
> really talking about is:
>
> 1.  Custom input
> 2.  Custom output

Yep, that summarizes it well.


> TAP:: might be a rather poor namespace because if someone wants to  
> read POSIX test results, they can't.  And we can't even *dream* of a  
> mixture of those.  But let's ignore the namespace for just a  
> moment.  Given rather confused state of the testing world in regards  
> to standardizing output, we have a unique opportunity to:
>
>  TAP::Parser::Grammar::TAP
>  TAP::Parser::Grammar::Ant
>  TAP::Parser::Grammar::POSIX
>  TAP::Parser::Grammar::TestNG
>  ... and so on


Agreed on the namespace, that aside: so you're suggesting having  
pluggable grammars as well?

That complicated things.  I can see the benefit though.

I've not tried writing one, but we'd need to make sure it was as easy  
as customizing in/outputs.  Each grammar would either need to have  
it's own source factory, or you'd need a way of mapping source to  
grammar.


> Use cases:
>
> 1.  Test::Builder 2.0 will have (iirc) support for outputting POSIX  
> test results.
> 2.  Hudson and other build systems typically use ant test results  
> (as far as I understand, jUnit doesn't really have any output  
> format.  The "jUnit XML" is really generated by ant. I might be  
> rather mistaken).

I'd love to either have a Hudson plugin for TAP, or be able to output  
something that Hudson groks.. We make extensive use of Hudson at work.


> Currently, we'd only need support for the TAP grammar, but  
> supporting the other one's would give testers a field day.  And once  
> parsing is normalized, custom output becomes the killer feature.  I  
> could see many larger testing systems switching to this.
>
> The problem would be a test suite which has combinations of POSIX,  
> TAP, etc.
>
>
> The custom "sources" originally mentioned would actually just be  
> ways of streaming appropriate test results into the grammar engine.

All makes sense.  Obviously would take some time though.  I think the  
questions we need to answer now are:

1. Do we want to go down this route?
2. Does this mean we should use a generic plugin system?

I'm happy saying yes to (1), I'm still not so sure about (2).


> And we still need test histories to round this out.  My  
> App::Prove::History on github (they're down right now, so I can't  
> link) is almost done, but it's very hackish internally because  
> App::Prove really didn't support it too well and because I didn't  
> understand the problem space well enough when I wrote it.  If you  
> want a commit bit, drop me a line.  I'll happily give you one :)

Test histories are definitely a good thing.  Currenty we rely on  
Hudson for 'em.

I'd like to take a look at this, but I can only bite off so much.. So  
unfortunatelt not right now.


>
>
> Cheers,
> Ovid
> --
> Buy the book         - http://www.oreilly.com/catalog/perlhks/
> Tech blog            - http://use.perl.org/~Ovid/journal/
> Twitter              - http://twitter.com/OvidPerl
> Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
>
> _______________________________________________
> tapx-dev mailing list
> tapx-dev at hexten.net
> http://www.hexten.net/mailman/listinfo/tapx-dev
> cpan: http://search.cpan.org/dist/TAP-Parser/
> bugs: http://rt.cpan.org/Public/Dist/Display.html?Name=TAP-Parser