[tapx-dev] Automated test report for TAP::Parser r226

[Andy Armstrong]

On 14 Aug 2007, at 19:20, Michael Peters wrote:
> That's nice idea. I'd like to see some sort of merge between  
> Test::SmokeStack
> and TestRunner::Multi (I like the Test::SmokeStack name better :)  
> so you can
> easily run multiple branches of multiple projects against multiple  
> interpreters.
> And then either watch the results, get them in an email, or send  
> them to Smolder.

Yes^3 :)

>> The architecture is that you specify a number of test cases in the
>> config file and an intepreter set to run them against. The test  
>> results
>> are gathered and sent to 0 or more formatters which do things like
>> generate an email, archive the results in a database &c.
>>
>> Thinking about it last night I convinced myself that instead of  
>> building
>> multiple formatters into SmokeStack I should just emit TAP and write
>> formatters that consume TAP.
>
> I'm a little worried about making TAP try to do too much. But maybe  
> that's just
> feature-creep paranoia. One problem I ran into when trying to use  
> just TAP for
> the uploads to Smolder was trying to capture meta data about the  
> whole run of
> the test suite itself and not individual test files. Like how long  
> did the tests
> take to run? Which order did the files run in?

Indeed. I think we need to allow YAML blocks to be associated with  
the test as a whole rather than just individual assertions to get  
round that. Oops - another feature.

>
>> At the end of a SmokeStack run I'd like to output a single TAP  
>> document
>> that describes the whole run - which brings me back to my proposal to
>> allow nested blocks in TAP[1]. Unfortunately the debate around that
>> last time caused a lot of traffic on the perl-qa list without leading
>> to anything.
>
> That's why I just did the archive thing for Smolder. There seemed  
> like a lot of
> inertia that I didn't have the tuits or clout to fight. So I did  
> something that
> solved my problem even if it's not the optimal way.
>
> The archive idea is nice because it also doesn't manipulate the TAP  
> output from
> the tests in anyway. It's the exact raw output from those test files.

I'm sure the archive idea is an excellent solution - it's obviously  
the best way to do it if you can't modify TAP syntax :)

>> On one hand it might be a little late to be adding features to  
>> TAP; on
>> the other we're not sure when we're next going to have an  
>> opportunity to
>> make such a major change.
>
> I suggest we keep pushing for TAP blocks. It may take some time and  
> effort, but
> would be worth it. In the meantime I'd like to go ahead with the  
> archive idea.
> It won't interfere with blocks if/when we get them and could still  
> be useful as
> a separate feature.

Sure, absolutely.

-- 
Andy Armstrong, hexten.net