Test::Harness features

[Steve Purkis]

On 9 Jun 2009, at 20:02, David Golden wrote:

[snip! dragons & use cases :)]

> From looking at my use case brainstorm, there are two general cases:
>
> (a) developers who want to use some advanced harness feature without
> messing around with the build tool
>
> (b) users who want ordinary module testing to do something other than
> run all tests, tee output to the screen and see an on-screen summary
>
> Even assuming that CPAN, CPANPLUS, M::B, EU::MM started implementing
> conditional support for TAP::Harness instead of Test::Harness, there's
> a synchronization gap between when/where new harness features get
> implemented and when those tools provide support.
>
> Thus -- I think there needs to be a general system for extending
> Test::Harness that goes beyond just App::Prove plugins.


I agree on the general system for extending *TAP*::Harness - that's  
what I was trying to get at in my email yesterday.  Not so sure about  
*Test*::Harness though, but more on that in a sec.

Start with extending TAP::Harness.  From what I can tell from your use  
cases, my conversations with David Wheeler, and my own experience, we  
have two main use cases to cater for in a plugin system:

1. Custom formatting (Archive, HTML, JSON, File, etc.)
2. Custom sources (Archive, db, File, Nested TAP?, PHP, etc.)

For anything more complicated, I think we could safely say 'sub-class  
TAP::Harness for $really_cool_feature'.  Does that make sense?

Typically (1) has been done with App::Prove plugins and/or custom  
TAP::Harness sub-classes.  Until now (2) has only been possible with  
custom TAP::Harness sub-classes.  I think all we're really proposing  
here is:

1. We let TAP::Harness load formatter & source 'plugins'.
2. We consider how to make this easily available to users via M::B,  
EU::MM et. al.
3. We set 1 & 2 as pre-requisites for the move into Core.

That should be fairly easy to achieve since half the battle is already  
fought with the source plugin stuff we've done - one of David's  
requirements was for it to be easily accessible via an existing config  
interface in M::B (I haven't tried EU:MM yet, I must admit).  If  
people are happy with that, I'm sure we could take the same approach  
for formatters.  App::Prove's plugins could then either stay as-is, or  
be adjusted to fit around the TAP::Harness plugin system if need be.

What if we decide that formatter & source plugins are not enough?   
Then we just create a general plugin system for TAP::Harness and make  
the sources & formatters special cases of that.

As for worrying about how this can be used via Test::Harness I'd say:  
if you want this feature, too bad, you've gotta upgrade.  Yes we need  
to worry about backwards compat, but as far as I know Test::Harness  
never supported plugins ;-).  And as it stands, I can kind of see the  
benefit of making back-doors with %ENV vars, but what stands out to me  
is the cost to us: added complexity, more complicated support cases, a  
stickier backwards compat web, and basically time that could be better  
spent.  So my question is: what's wrong with asking people to upgrade?

This of course means we'd need to ensure there was a clear upgrade  
path with deps across all the affected modules (T::H, M::B, EU::MM,  
and CPAN* if need be).  But I'm assuming that was the plan anyway...

my 2¢,
-Steve