Re: benchmarking Flex practices

John Naylor <john(dot)naylor(at)2ndquadrant(dot)com> writes:
> [ v4 patches for trimming lexer table size ]

I reviewed this and it looks pretty solid. One gripe I have is
that I think it's best to limit backup-prevention tokens such as
quotecontinuefail so that they match only exact prefixes of their
"success" tokens. This seems clearer to me, and in at least some cases
it can save a few flex states. The attached v5 patch does it like that
and gets us down to 22331 states (from 23696). In some places it looks
like you did that to avoid writing an explicit "{other}" match rule for
an exclusive state, but I think it's better for readability and
separation of concerns to go ahead and have those explicit rules
(and it seems to make no difference table-size-wise).

I also made some cosmetic changes (mostly improving comments) and
smashed the patch series down to 1 patch, because I preferred to
review it that way and we're not really going to commit these
separately.

I did a little bit of portability testing, to the extent of verifying
that the oldest and newest Flex versions I have handy (2.5.33 and 2.6.4)
agree on the table size change and get through regression tests. So
I think we should be good from that end.

We still need to propagate these changes into the psql and ecpg lexers,
but I assume you were waiting to agree on the core patch before touching
those. If you're good with the changes I made here, have at it.

regards, tom lane