Parallel thoughts

[Andy Armstrong]

On 2 Oct 2007, at 03:37, Eric Wilhelm wrote:
>> The problem with 2 is that all the parsing is still happening in a
>> single process. This bottleneck gets worse as you throw more cores at
>> the problem.
>
> This "drawback" is also a benefit if you look at it WRT callbacks and
> other things not having to deal with the forks and/or storable.

That's a benefit in the way that only running a single process is a  
benefit because it's easier :)

> The problem with #3 is that you cannot distribute it across machines
> without having (matching) Storable and TAP::Parser installed on the
> nodes.  There's probably also something to do with running a different
> perl version as the harness vs the tests.

Yes, but the problem with the other two is that they're CPU bound  
before we've done anything interesting.

> If you look at it from the POV of parsing an archive (rather than
> running a harness) I would think multi-threading would be the way  
> to go
> to speed-up the parsing process, otherwise I would guess that IPC
> becomes a real PITA and everything gets really complicated.  Of  
> course,
> it would be nice to just make the grammar run faster and quit  
> trying to
> throw cores at it.

We haven't spent much of our total CPU time just parsing archives so  
far. I'm guessing you don't like the IPC / fork based version?

> Why is the multiplexer not a subclass of TAP::Harness?

The multiplexer is a separate class. All three harness  
implementations are next to each other in TAP::Harness at the moment  
for ease of experimenting with.

-- 
Andy Armstrong, Hexten