[tapx-dev] Automated test report for TAP::Parser r226
andy at hexten.net
Tue Aug 14 19:28:12 BST 2007
On 14 Aug 2007, at 19:20, Michael Peters wrote:
> That's nice idea. I'd like to see some sort of merge between
> and TestRunner::Multi (I like the Test::SmokeStack name better :)
> so you can
> easily run multiple branches of multiple projects against multiple
> And then either watch the results, get them in an email, or send
> them to Smolder.
>> The architecture is that you specify a number of test cases in the
>> config file and an intepreter set to run them against. The test
>> are gathered and sent to 0 or more formatters which do things like
>> generate an email, archive the results in a database &c.
>> Thinking about it last night I convinced myself that instead of
>> multiple formatters into SmokeStack I should just emit TAP and write
>> formatters that consume TAP.
> I'm a little worried about making TAP try to do too much. But maybe
> that's just
> feature-creep paranoia. One problem I ran into when trying to use
> just TAP for
> the uploads to Smolder was trying to capture meta data about the
> whole run of
> the test suite itself and not individual test files. Like how long
> did the tests
> take to run? Which order did the files run in?
Indeed. I think we need to allow YAML blocks to be associated with
the test as a whole rather than just individual assertions to get
round that. Oops - another feature.
>> At the end of a SmokeStack run I'd like to output a single TAP
>> that describes the whole run - which brings me back to my proposal to
>> allow nested blocks in TAP. Unfortunately the debate around that
>> last time caused a lot of traffic on the perl-qa list without leading
>> to anything.
> That's why I just did the archive thing for Smolder. There seemed
> like a lot of
> inertia that I didn't have the tuits or clout to fight. So I did
> something that
> solved my problem even if it's not the optimal way.
> The archive idea is nice because it also doesn't manipulate the TAP
> output from
> the tests in anyway. It's the exact raw output from those test files.
I'm sure the archive idea is an excellent solution - it's obviously
the best way to do it if you can't modify TAP syntax :)
>> On one hand it might be a little late to be adding features to
>> TAP; on
>> the other we're not sure when we're next going to have an
>> opportunity to
>> make such a major change.
> I suggest we keep pushing for TAP blocks. It may take some time and
> effort, but
> would be worth it. In the meantime I'd like to go ahead with the
> archive idea.
> It won't interfere with blocks if/when we get them and could still
> be useful as
> a separate feature.
Andy Armstrong, hexten.net
More information about the tapx-dev