UNKNOWN despite only failing tests -- how come?

[David Golden]

> > * test.pl, no output, exit 0 -> EU::MM says pass, M::B says unknown
> > * test.pl, no output, exit 1 -> EU::MM says fail, M::B says unknown

For clarity, let me add the following cases (where T::H < 3.05 includes 2.XX):

* foo.t, no output, exit 0 -> T::H < 3.05 says unknown, T::H 3.05 says fail
* foo.t, no output, exit 1 -> T::H < 3.05 says unknown, T::H 3.05 says fail

The last one is the bug that T::H 3.05 fixed.  That's a good fix, I
think, but it also hit the exit 0 case, depending on whether one
thinks that should be unknown or fail.

> T::H currently works on the premise that if you run a test and it
> outputs nothing it's failed. I guess we could special-case test.pl but
> that's messy - would be good if we could avoid that.

I think it's too late for that.  I've done some more testing and I
think we have three options:

(a) Keep T::H 3.05 behavior -- CPAN::Reporter adds special-case tests
for T::H  < 3.05 and T::H >= 3.05

(b) Change T::H to say "unknown" for the no output, exit 0 case
(regardless of whether it's test.pl or not) -- CPAN::Reporter adds
special-case tests for T::H < 3.05, = 3.05, > 3.05.

(c) Change T::H to say "unknown" for the no output, exit 0 case, but
only for test.pl -- CPAN::Reporter adds special-case tests for T::H <
3.05, = 3.05, > 3.05.

So at this point, I've written a new test file for T::H
version-specific outcomes, since that seems inevitable thanks to the
legitimate 3.05 bug fix.

I just need to know whether the Perl collective thinks that (b) or (c)
make sense so I know whether to add the appropriate logic for T::H >
3.05.

> > Overall, I'd prefer if Test::Harness releases that changed the
> > definition of failure were released in a developer version first so we
>
> I'm really sorry about that David. It was a dumb thing to do - no
> excuses. I'll make sure it doesn't happen again.

That's OK.  When I've got bug fixes, I tend to want to get them out
the door too, and CPAN::Reporter tests take a while.  On the positive
side, I realized that I needed to add a test case for foo.t, no
output, exit 1.

I'm writing this while I test the CPAN::Reporter trunk on T::H 3.05,
T::H 3.04 and T::H 2.64.  (I've had do that enough, and under each of
M::B and EU::MM that I really need to automate it.)  If all goes well,
I'll check in the code and you can pull the latest for testing from
the repository:

http://dagolden.googlecode.com/svn/CPAN-Reporter/trunk/

If it has t/56_test_report_harness_versions.t then you've got the
version with the fixes.  If you don't see that, then I'm still working
on it.

David