Re: Have we out-grown Flex?

On Tue, May 1, 2012 at 8:53 PM, Peter Geoghegan <peter(at)2ndquadrant(dot)com> wrote:
> I'm certainly not suggesting that this isn't something that, in order
> to be adopted, will have to be carefully considered from many angles,
> of which the performance of the resulting lexer is only one, but at
> the same time I dare say that this approach is the "low-hanging fruit"
> of optimising for the use-case where parser overhead is unreasonably
> high. Am I barking up the wrong tree? At the very least, hopefully
> these simple observations will provoke thought - Flex/Lex is not the
> only tool for generating yacc-compatible lexical analysers, and it may
> not be the best one for our current needs.

I think Tom's question of whether the parser or lexer is the problem
is something that ought to be investigated. Personally, I suspect
that our tendency to use lists everywhere, for everything, an artifact
of our proud LISP heritage, may be costing us dearly in terms of parse
time. However, there's a difference between suspicion and proof, and
I certainly have none of the latter.

One fairly major obstacle to using quex for anything is that it
doesn't appear to be packaged on either Fedora or Ubuntu (I tried
apt-cache search quex on my Ubuntu laptop, and yum install quex on an
F16 machine). 'port install quex' on my MacOS X doesn't find it
either. This means that if we were to switch to quex, the barrier to
compiling PostgreSQL would go up very significantly. I realize
there's a chicken-and-egg problem there, since if quex is awesome and
nobody uses it because it isn't packaged, then it will never become
popular enough for anyone to consider packaging it. Nonetheless, I'm
not sure I want to be the guinea pig. Even if the actual
patch-as-committed-to-git to make the switch to quex were not that
large, the amount of follow-on work we'd be creating for every
developer and packager of PostgreSQL would be quite formidable. We
shouldn't do that unless it's pretty much awesome.

One possible way forward would be to try to be compatible with both
quex and flex. We'd need a buildfarm critter or two running with quex
to detect breakage there, though, since a lot of people would probably
continue to run flex. But I'm not entirely sure it's worth doing even
that much unless we can show a worthwhile performance bump. The core
scanner doesn't change much, so "more features" isn't too compelling
as a reason for change, especially if we need to remain
flex-compatible.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company