[tap-l] Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 15:38, Michael Peters wrote:
> Isn't this the problem that SKIP was designed to fix? If it needs to  
> dynamically
> determined if a test should run, isn't the test itself the best  
> place to do
> that? xt tests shouldn't be run unless explicity done so by the  
> runner of the
> tests (a developer) who presumably knows that they are doing.

The feature request that started all this relates to a large test  
suite most of which needs to be skipped if a graphical environment  
isn't available - so presumably it'd involve executing hundreds of  
tests just to find out that they skipped.

> Instead of "something else" telling the harness which tests to run,  
> why not have
> the harness run them all and the individual tests can decide if they  
> don't want
> to go through with it or not. If "something else" tells the harness  
> which tests
> to run, we loose the information about which tests didn't run and why.

Having to run each test to find out it didn't want to run doesn't  
scale well. Having a mechanism to dynamically select subsets of tests  
can only encourage people to write more tests I'd have thought.

If we decide on a set of rules for which xt/* tests are executed under  
which circumstances it also gives us a language in which to express  
those rules - so instead of being hard wired into TAP::Harness we'll  
have a little TSP document that could be used with any TAP parser.

I've also seen a discussion of test ordering and the practice of  
numbering tests to guarantee a ordering recently; ordering and more  
complex dependencies could also be handled by a t/controller.t that  
emits TSP.

> Testing in Perl is really nice and simple and I'm not seeing the  
> benefits or the
> problems being solved with this complexity that can't already be  
> solved using
> something else.


You could presumably have said the same about the idea of TAP based  
testing. You can after all just have a bunch of programs that pass or  
fail and run them from the shell.

I'm being slightly facetious there :)

I agree that parsimony should guide our decisions but I'm not finding  
it hard to think of use cases for TSP.

-- 
Andy Armstrong, Hexten