[tapx-dev] benchmark

[Ovid]

--- Eric Wilhelm <scratchcomputing at gmail.com> wrote:

> # from Eric Wilhelm
> # on Saturday 28 July 2007 06:52 pm:
> 
> >Ouch.
> 
> Still ouch.
> 
> >I know we're doing more things correctly, but I'm hoping to find
> >something simple.  It's usually case.
> 
> It doesn't look like correctness is costing us that much.  Rather, it
> 
> looks like it is the abstraction and a *lot* of little things.

A few months ago, when I felt that TAP::Parser (then TAPx::Parser) was
relatively stable, I started to make performance enhancements and a few
worked their way in, though I was not too terribly aggressive about it.
 If you feel that we can make significant performance improvements,
that would be welcome.  If you feel that they obscure the intent or
maintainability of the code, comments would be most welcome.

Right now, there's a certain amount of performance which must be
sacrificed for the much richer information we capture.  That being
said, I would *love* to at least come close to the performance of prove
and Test::Harness and would be willing to accept some unpleasantness in
the code for this win.  The only caveat I would add is to ensure that
we do not break backwards compatibility (though I guess that probably
didn't need to be said.

In short, please feel free to make commits which improve the overall
performance.  I trust your judgment.  
 
> Ordering the grammar tokenizer regexps as 'qw(test comment yaml
> bailout 
> plan version)' (as opposed to 'values(%grammar)') seems to help too.

Damn.  I had tried that once, but it didn't seem to make much of a
difference.  Maybe the code has changed that much or I should have had
better benchmarks :)
 
> For instance, TAP::Parser::_aggregate_results() is a big slow-down,
> but 
> the only way to get it out of the inner loop is to wait to do the 
> aggregation later.  That means just holding the result objects.  E.g.

> do we really need to know 'failed' and 'actual_failed', etc on the
> fly?

This could require a slight API change.  We explicitly do not hold
those objects because infinite TAP streams are valid.  I doubt many
people have enough computer memory to handle this.  What size test
suite would someone need to run before this becomes problematic?  I
couldn't say, but it's something to bear in mind.

To make this work, both TAP::Harness and TAP::Harness::Compatible would
need some way of saying "cache results".

Cheers,
Ovid

--
Buy the book  - http://www.oreilly.com/catalog/perlhks/
Perl and CGI  - http://users.easystreet.com/ovid/cgi_course/
Personal blog - http://publius-ovidius.livejournal.com/
Tech blog     - http://use.perl.org/~Ovid/journal/