[tapx-dev] TAP::Parser recursive tests

[Joshua Keroes]

Here's the conversation Ovid and I just had about handling
subdirectories off t/, reformatted and clarified for email posterity:

Joshua Keroes: If I understand your email correctly, it appears as if
TAP::Parser (and by extension, MB) takes this issue into account. If
we don't make this the default behaviour, we can punt on the
IPC::Singleton detail. That becomes a nonissue

ovidperl: It's just a matter of specifying recursive tests.  It cannot
be the default behavior because that breaks backwards compatibility
for TH3

Joshua Keroes: I do, however, believe it would behoove the MB folks to
add do_recursive_tests => 1 or whatever the option is [as a default].
At the very least, document it better

ovidperl: They have something like that, but I can't recall what.

Joshua Keroes: I went through their docs and it's not well documented;
not at all. Let's see. There was one more detail in the email...
right, the "do we ignore certain test sub directories" question

ovidperl: That sounds like a feature we can use for author_tests.

Joshua Keroes: [As for the] question of whether we should ignore
certain subdirectories of modules: I believe, that since runtests() is
typically invoked with a list of files instead of one or more
directories, it may be a nonissue. That said, there are several
modules on CPAN which have demo/ or example/ dirs full of test files.

Joshua Keroes: Without writing extra code, the user can solve this
problem by calling runtests( qw[t/ok1 t/ok2 t/ok3] ); and not listing
t/demo

ovidperl: Have you seen the Perl-QA issue regarding producer and
consumer tests? A module should fail to install if it's "mangle_foo"
function fails its tests, but it should not fail to install if the POD
fails a test.

Joshua Keroes: I like that differentiation. And while I haven't seen
that issue, I applaud it.

ovidperl: Right, but ease of use should be the first target.  By
default, a user should never have to worry about producer tests.

Joshua Keroes: I've seen that implemented with SKIP, as in skip these
tests unless I'm the developer.

ovidperl: That means that if they do everything the way they've always
done things before, things just work. I don't expect the end user to
change their behavior. That might be a decent workaround.

Joshua Keroes: *nod*

__END__

On 7/30/07, Eric Wilhelm <scratchcomputing at gmail.com> wrote:
> # from Joshua Keroes
> # on Monday 30 July 2007 07:13 pm:
>
> >my $build = Module::Build->new(
> >...
> > test_files => [
> > glob('t/*.t'), glob('t/*/*.t'),
> > glob('t/*/*/*.t'), glob('t/*/*/*/*.t'),
> > glob('t/*/*/*/*/*.t')
> > ]
> >);
>
> That's not needed in any version of Module::Build that I know of.  Just
> "recursive_test_files => 1".
>
> And that same functionality can be gotten with:
>
>   runtests -r -l
>
> (which implies "t/".)
>
> >Another way other modules have implemented this is to write a single
> >test file (e.g. test.pl) that calls runtests() on the *.t files in
> >subdirectories.
>
> What we can't do right now (I think) is *not* run *.t files in t/ if the
> -r flag is given.  Do we need that?
>
>   t/run_the_other_tests.t
>   t/run_these/foo.t
>   t/other/foo.t
>   t/other/bar.t
>
> That is, if runtests is expected to run *only* "t/run_the_other_tests.t"
> and "t/run_these/foo.t", they have to be explicit arguments.
>
>   runtests -l t/run_the_other_tests.t t/run_these/*.t
>
> or, like
>
>   runtests -l $(find t -name '*.t' | grep -v '/other/')
>
> (where run_the_other_tests.t runs everything in "other/")
>
> >Question: do we make this (-r) the default behavior or not? I'd like to
> >say "yes" if we can successfully workaround the above modules'
> >homegrown testing solutions.
>
> I'll second that, but it does need to address the fact that `prove`
> never did this, this means "one big fat assumption" which has led to
> the workarounds you've sighted :-D, and cited (trimmed.)
>
> >Thought: modules that have already implemented recursive testing might
> >inadvertently run the tests more than once. That is, we find all the
> >*.t files and run them, then their custom 00_recusively_test.t does
> >the same thing. Solution is to tack some logic on the TAP::Harness
> >singleton (it is a singleton, right?).
>
> The trouble is that the test runs in a forked process.  IPC::Singleton?
>
> --Eric
> --
> perl -e 'srand; print join(" ",sort({rand() < 0.5}
>   qw(sometimes it is important to be consistent)));'
> ---------------------------------------------------
>     http://scratchcomputing.com
> ---------------------------------------------------
>
>