Re: scanner/parser minimization

On 2 March 2013 18:47, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> wrote:

>> Uh ... no. I haven't looked into why the flex tables are so large,
>> but this theory is just wrong. See ScanKeywordLookup().
>
>
> Interestingly, the yy_transition array generated by flex used to be much
> smaller:
>
> 8.3: 22072 elements
> 8.4: 62623 elements
> master: 64535 elements
>
> The big jump between 8.3 and 8.4 was caused by introduction of the unicode
> escapes: U&'foo' [UESCAPE 'x'] . And in particular, the "error rule" for the
> UESCAPE, which we use to avoid backtracking.
>
> I experimented with a patch that uses two extra flex states to shorten the
> error rules, see attached. The idea is that after lexing a unicode literal
> like "U&'foo'", you enter a new state, in which you check whether an
> "UESCAPE 'x'" follows. This slashes the size of the array to 36581 elements.

+1 to do this sooner rather than later

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services