The whole moving tests thing

[Michael G Schwern]

So I'm talking with Ovid about these POD tests on IM and why I think it's such
a bad idea over a fairly small thing.  And he's right, I am making a big
kerfuffle about a small thing, but hating optional tests is a pet peeve of mine.

I suppose it boils down to this:

1)  It adds complexity to a project which does not yet need that complexity.

Eric's concerns about POD incompatibilities aren't.  This plays out from my
experiences with Test::More, ExtUtils::MakeMaker and Test::Harness.  Yes,
Test::Harness already has Test::Pod tests.

Maybe TAP::Parser will need it in the future, but not now.  Not for =head3.


2)  Optional tests don't get run.  You forget.  This means real bugs slip in.

It's not as simple as "just move the tests into another directory".  How do
they get run?  Will we remember to run them?  No, people will forget.  That's
why I say an optional test might as well be deleted.  If TAP::Parser had an
automatic smoke server that would mitigate this.

#2 makes it a trade-off.  On the one hand, you avoid false test failures for
the users but at the expense of making your tests overall less effective.

#2 is why I say you might as well just delete optional tests and tests which
are "ok to fail".


3)  It starts a slippery slope down which TAP::Parser does not yet have to go.

Optional tests and tests for which failure isn't really failure, this blurs
the meaning of the test results.  You start to get into decisions over whether
or not a test should be "ok to fail" or not.  Like allowing just one failing
test or just one warning, it erodes confidence in the test suite.

Because of #1 this is not a slope TAP::Parser has to start down yet.


4)  Those user test failures are actually useful.

Some failures are spurious.  Some are real.  Maybe you did use some POD that
doesn't display correctly on 5.6, but since all the devs use 5.8 nobody knew.
 A test failure from a user lets you fix that.  It's the best way to get
feedback and fix the problem.

Again, a trade off.  A single test tests many things.  By portioning out some
only to be run by the developers you invite homogeny and a "works on my
machine" problem.


So there you go.  In the end, I'm not doing much coding on the project anyway
so what I want doesn't matter.  It just bothers me when its for the wrong reasons.


-- 
The interface should be as clean as newly fallen snow and its behavior
as explicit as Japanese eel porn.