Simplistic tokenizer cheat?

[Andy Armstrong]

On 15 Sep 2007, at 00:00, Michael G Schwern wrote:
> Can we put in a simplistic tokenizer cheat to bypass the more  
> expensive full
> grammar check?
>
> 	if( $line =~ /$simple_regex/ ) {
> 		proceed directly to parsing
> 	}
> 	else {
> 		full tokenizer
> 	}

+1 if it's a performance win.

-- 
Andy Armstrong, hexten.net