Machine readable test summary
xdaveg at gmail.com
Tue Sep 4 03:31:42 BST 2007
On 9/3/07, Andy Armstrong <andy at hexten.net> wrote:
> If I'm going to produce a patch for CPAN::Reporter I'd
> rather say "here's a shiny, stable new interface that avoids the need
> for screen scraping" rather than "I updated your code so that it
> supports the latest Test::Harness - but it's still as fragile as hell".
I pretty much knew that the new Test::Harness would break things --
one reason why I raised the issue in the first place, but I figured
that I'd wait for the alpha before I started changing things.
There are two sources of fragility in Test::Harness -> CPAN::Reporter
1) Test::Harness failure categories don't cleanly map to CPAN Testers
categories -- e.g. CPAN::Reporter really needs to know if "FAILED" was
due to "no tests" run/provided, which is a CPAN Testers unknown.
Therefore, a purely binary exit status is useless.
2) String scraping -- for which I don't think I need to elaborate
if Test::Harness put out some sort of standard result code that was
easy to parse and that aligned better with CPAN Testers categories,
these points of breakage would go away. That could be something
*added* to current messages if that's easier to do than other means.
Even something like an HTTP response code would be fine. E.g. 2\d\d
means pass, 3\d\d means unknown, and 4\d\d means fail (and maybe 1\d\d
means NA, if Test::Harness can even identify those).
CPAN::Reporter doesn't really need more detailed test diagnostics at
the current time -- and unless there was a widespread desire to
capture those on CPAN::Testers, I'd rather not go mucking about with
environment variables and files and the rest -- it seems like
unnecessary complexity for what CPAN::Reporter needs.
More information about the tapx-dev