Machine readable test summary
xdaveg at gmail.com
Tue Sep 4 14:53:43 BST 2007
On 9/4/07, Andy Armstrong <andy at hexten.net> wrote:
> As a matter of interest David did you find the exit code from make
> test to be an unreliable indication of passed / failed? I'm working
> on running tests for Test::SmokeStack so I'm interested to know if
> there are any pitfalls in trusting the exit code.
I'd have to check my test cases. I think the nasty edge cases involve
test.pl and the fact that EU::MM and M::B treat it differently.
EU::MM runs test.pl directly and takes its exit code as success or
failure. M::B passes test.pl to Test::Harness and gets that result
While the way in which they report failure differs, most cases give
the same overall result. Where I have found a split in behavior is
where t/*.t tests pass and test.pl exits with exit code 0 but doesn't
output any TAP. EU::MM treats that as success but M::B doesn't.
Generally, I think that the make exit code is reliable for the binary
pass/fail, but won't help distinguish the "NOTESTS" case.
I used to not be able to use the exit code from make directly because
CPAN::Reporter tees the output and the make exit code is therefore
lost. However, later versions of CPAN::Reporter wrap any system calls
into another script that prints the exit code to STDOUT -- another
ugly hack to do IPC via STDOUT. Using that I could probably now just
rely on the exit code of make, but I haven't needed to do that to pass
any existing test cases so I haven't bothered.
I may look into that again to resolve the situation of recursive
Makefiles, which has been sitting on the todo list for a while. I'd
rather rely on the exit code than try to parse out success/failure
from the output of individual components.
More information about the tapx-dev