App::Prove::State Questions

[Andy Armstrong]

On 21 Aug 2008, at 20:17, Ovid wrote:
> I'm trying to improve the interface to App::Prove::State.  My short  
> term goal is to create App::Prove::State::SQLite to allow people to  
> store test suite information over time.
>
> This raises some questions about App::Prove::State internals.
>
> Issues.
>
> 1.   There's a DESTROY method.  This saves the state even if I kill  
> the test suite.  This can throw long term test suite results off,  
> but I can mark a test suite as 'incomplete' to get around this.

Yeah, I guess it shouldn't do that.

> 2.  There's a 'last_result' key.  This is true if the test program  
> failed or had problems (such as passing TODO tests).  I'd like to  
> split that key.

Yup, sounds good.

> 3.  Of course, I intend to extend the keys to capture tests run,  
> tests failed and so on.

Yup.

> 4.  Tracking reorganization of the test suite seems problematic.  If  
> people rename or move test programs, this can throw results off, but  
> we currently have no way of uniquely identifying a test program  
> (much less an individual test).

Yeah, in the absence of a unique ID for each assertion I think we'd  
need to use Algorithm::Diff on a normalised version of the results (no  
test numbers) to track them heuristically. It'd probably work fairly  
well in most cases but large re-organisations (renaming -and- moving  
tests in one move) would likely fool it.

Something like

App::Prove::State (in Test::Harness)
  \-- App::Prove::State::Tracker (on CPAN)
       |-- App::Prove::State::SQLite
       ....
       |-- App::Prove::State::MySQL
       \-- App::Prove::State::Pg

perhaps? Where A::P::S::Tracker is responsible for tracking the  
movement of tests over time.

And perhaps we want to standardise a syntax for embedding a UID for an  
assertion in its description? If we did that then A::P::S::Tracker can  
use that information if it's available.

> Am I likely to break anything?  I don't know if anyone is really  
> using the state syntax.


I don't /think/ so. The file format is undocumented so I don't think  
people can be too surprised if it changes - and I'm not aware of any  
3rd party apps apart from your script to report on test execution times.

-- 
Andy Armstrong, Hexten