Machine readable test summary

[David Golden]

On 9/4/07, Andy Armstrong <andy at hexten.net> wrote:
> As a matter of interest David did you find the exit code from make
> test to be an unreliable indication of passed / failed? I'm working
> on running tests for Test::SmokeStack so I'm interested to know if
> there are any pitfalls in trusting the exit code.

I'd have to check my test cases.  I think the nasty edge cases involve
test.pl and the fact that EU::MM and M::B treat it differently.
EU::MM runs test.pl directly and takes its exit code as success or
failure.  M::B passes test.pl to Test::Harness and gets that result
back.

While the way in which they report failure differs, most cases give
the same overall result.  Where I have found a split in behavior is
where t/*.t tests pass and test.pl exits with exit code 0 but doesn't
output any TAP.  EU::MM treats that as success but M::B doesn't.
Generally, I think that the make exit code is reliable for the binary
pass/fail, but won't help distinguish the "NOTESTS" case.

I used to not be able to use the exit code from make directly because
CPAN::Reporter tees the output and the make exit code is therefore
lost.  However, later versions of CPAN::Reporter wrap any system calls
into another script that prints the exit code to STDOUT -- another
ugly hack to do IPC via STDOUT.  Using that I could probably now just
rely on the exit code of make, but I haven't needed to do that to pass
any existing test cases so I haven't bothered.

I may look into that again to resolve the situation of recursive
Makefiles, which has been sitting on the todo list for a while. I'd
rather rely on the exit code than try to parse out success/failure
from the output of individual components.

Regards,
David