[tapx-dev] Automated test report for TAP::Parser r226

[Michael Peters]

Andy Armstrong wrote:

> That's fine thanks. I checked out the latest version from Sourceforge.
> I'm happy to host it on my box if I can work out how :)

I don't (or rather, PlusThree doesn't) mind hosting it. In fact it's not the
hosting of Smolder that is the hard part. It's the setting up of the automated
test runner.

> Yes. I've been thinking about this quite a bit. The little smoke.pl
> script has turned into Test::SmokeStack (not released) which is a pretty
> generic framework for running tests against multiple versions of
> interpreters - not just Perl.

That's nice idea. I'd like to see some sort of merge between Test::SmokeStack
and TestRunner::Multi (I like the Test::SmokeStack name better :) so you can
easily run multiple branches of multiple projects against multiple interpreters.
And then either watch the results, get them in an email, or send them to Smolder.

> The architecture is that you specify a number of test cases in the
> config file and an intepreter set to run them against. The test results
> are gathered and sent to 0 or more formatters which do things like
> generate an email, archive the results in a database &c.
> 
> Thinking about it last night I convinced myself that instead of building
> multiple formatters into SmokeStack I should just emit TAP and write
> formatters that consume TAP.

I'm a little worried about making TAP try to do too much. But maybe that's just
feature-creep paranoia. One problem I ran into when trying to use just TAP for
the uploads to Smolder was trying to capture meta data about the whole run of
the test suite itself and not individual test files. Like how long did the tests
take to run? Which order did the files run in?

> At the end of a SmokeStack run I'd like to output a single TAP document
> that describes the whole run - which brings me back to my proposal to
> allow nested blocks in TAP[1]. Unfortunately the debate around that
> last time caused a lot of traffic on the perl-qa list without leading
> to anything.

That's why I just did the archive thing for Smolder. There seemed like a lot of
inertia that I didn't have the tuits or clout to fight. So I did something that
solved my problem even if it's not the optimal way.

The archive idea is nice because it also doesn't manipulate the TAP output from
the tests in anyway. It's the exact raw output from those test files.

> I'd really like to have this be valid TAP
> 
> 1..2
> begin 1 Test against 5.0.5
>      1..3
>      ok 1 perl Makefile.PL
>      ok 2 make
>      begin 3 Run tests
>          begin 1 t/000-load.t
>              1..4
>              ok 1 - use TAP::Parser;
>              ok 2 - ... and it should have the correct version
>              ok 3 - use TAP::Parser::Aggregator;
>              ok 4 - ... and it should have the correct version
>          end 1
>      end 3
> end 1
> begin 2 Test against 5.6.1
>      etc etc
> end 2

I think that would be a good solution too.

> That'd make it much easier to repackage output from tests into a valid
> TAP container document - you can encapsulate an existing TAP stream just
> by wrapping it in begin / end. No need to worry about patching up test
> numbering.
> 
> That's just one use case for blocks - but I'd really like it now for
> SmokeStack.
> 
> On one hand it might be a little late to be adding features to TAP; on
> the other we're not sure when we're next going to have an opportunity to
> make such a major change.

I suggest we keep pushing for TAP blocks. It may take some time and effort, but
would be worth it. In the meantime I'd like to go ahead with the archive idea.
It won't interfere with blocks if/when we get them and could still be useful as
a separate feature.

And if we do get TAP blocks then it would be really easy to allow Smolder (or
SmokeStack, or any other tool) to use them.

-- 
Michael Peters
Developer
Plus Three, LP