Re: Doing psql's lexing with flex

Tom Lane wrote:

>I got interested enough in the psql-with-flex problem to go off and
>solve it. Attached is a working patch, which I'm now debating whether
>to apply. Comments solicited...
>
>The patch removes about 200 lines of very spaghetti-ish code in
>mainloop.c. However, it adds an 875-line flex source file, which
>might be thought a bad tradeoff :-(. One bright spot is that about
>half of that total is a direct copy of the main backend lexer, so
>it's not really as much new, separately maintainable code as all that.
>Also, Andrew Dunstan's patch for supporting dollar-quoting would add
>about 100 lines to mainloop.c, versus only a dozen or so lines in the
>flex implementation.
>

Am I missing something, or is dollar quoting not in this patch? Perhaps
you have a followup patch?

>Once that's taken into account I don't think there
>is a lot of difference in effective SLOC to maintain. I'm also of the
>opinion that the new C code in psqlscan.l is much more straightforward
>than the code removed from mainloop.c, though having just written it,
>I'm no doubt pretty biased.
>
>Bruce was asking about speed. On normal-size queries I cannot measure
>any difference at all. For testing purposes I made up a file containing
>a single 750K query (just a "SELECT big-honking-string-constant", with
>the string literal broken into lines of 75 bytes). The client-side
>(psql) CPU time to run this file looks about like this on my machine:
>
> PGCLIENTENCODING
> UNICODE SJIS
>
>CVS tip 1.57 1.82
>
>flex implementation 0.93 2.33
>
>The flex implementation is consistently faster than CVS tip when dealing
>with backend-compatible encodings (such as UTF-8). It's consistently
>slower when it has to deal with a non-backend-safe encoding such as SJIS
>or Big5. But for real-world cases the differential is down in the noise
>either way.
>
>I'm inclined to apply this but I can see where a person not comfortable
>with flex might feel differently. Opinions?
>
>

In principle this is the Right Thing (tm).

We use flex elsewhere, of course, so the fact that it is a flex scanner
doesn't seem to me to be any sort of strong argument.

If we are going to do this we need to get it in ASAP, so it gets plenty
of beating on.

cheers

andrew