[tapx-dev] Tracking tests over time

[Ian Malpass]

Andy Armstrong wrote:

> With TAP that'd require either adding a persistent identifier to tests,
> requiring that the test numbering doesn't change over time (new tests
> added at the end) or that there's some heuristic that attempts to track
> assertions that have been renumbered.

Requiring that test numbering doesn't change seems like an awfully large 
burden. It would mean you couldn't add extra values to a loop with a 
test in it, for example (unless it happened to be at the end of your 
test). It would be difficult to enforce, easy to forget/ignore/not know 
about, and failure would be awkward.

A heuristic would be complicated and fragile, and if (when) it went awry 
your data would, again, go screwy.

As such, I think I'd favour the persistent identifier approach, and add 
an extra argument to each test assertion:

    is ( $got, $exptected, $test_name, $id );

You could then create and use a subclass of Test::More to care about the 
extra argument. Anyone just using plain Test::More wouldn't see any 
difference. It wouldn't be particularly taxing when writing tests, and 
would only impact those who felt that it would give them some benefit.

I'm not sure how you'd deal with not using Test::More (or that use 
Test::More internally), or using things that do stuff like 
all_pod_files_ok(), but those seem like solvable problems.

Ian