Skip site navigation (1) Skip section navigation (2)

Re: Doing psql's lexing with flex

From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: pgsql-patches(at)postgresql(dot)org
Subject: Re: Doing psql's lexing with flex
Date: 2004-02-18 00:50:22
Message-ID: (view raw, whole thread or download thread mbox)
Lists: pgsql-patches

Tom Lane wrote:

>I got interested enough in the psql-with-flex problem to go off and
>solve it.  Attached is a working patch, which I'm now debating whether
>to apply.  Comments solicited...
>The patch removes about 200 lines of very spaghetti-ish code in
>mainloop.c.  However, it adds an 875-line flex source file, which
>might be thought a bad tradeoff :-(.  One bright spot is that about
>half of that total is a direct copy of the main backend lexer, so
>it's not really as much new, separately maintainable code as all that.
>Also, Andrew Dunstan's patch for supporting dollar-quoting would add
>about 100 lines to mainloop.c, versus only a dozen or so lines in the
>flex implementation.  

Am I missing something, or is dollar quoting not in this patch? Perhaps 
you have a followup patch?

>Once that's taken into account I don't think there
>is a lot of difference in effective SLOC to maintain.  I'm also of the
>opinion that the new C code in psqlscan.l is much more straightforward
>than the code removed from mainloop.c, though having just written it,
>I'm no doubt pretty biased.
>Bruce was asking about speed.  On normal-size queries I cannot measure
>any difference at all.  For testing purposes I made up a file containing
>a single 750K query (just a "SELECT big-honking-string-constant", with
>the string literal broken into lines of 75 bytes).  The client-side
>(psql) CPU time to run this file looks about like this on my machine:
>CVS tip			1.57		1.82
>flex implementation	0.93		2.33
>The flex implementation is consistently faster than CVS tip when dealing
>with backend-compatible encodings (such as UTF-8).  It's consistently
>slower when it has to deal with a non-backend-safe encoding such as SJIS
>or Big5.  But for real-world cases the differential is down in the noise
>either way.
>I'm inclined to apply this but I can see where a person not comfortable
>with flex might feel differently.  Opinions?

In principle this is the Right Thing (tm).

We use flex elsewhere, of course, so the fact that it is a flex scanner 
doesn't seem to me to be any sort of strong argument.

If we are going to do this we need to get it in ASAP, so it gets plenty 
of beating on.



In response to


pgsql-patches by date

Next:From: Tom LaneDate: 2004-02-18 04:10:19
Subject: Re: Doing psql's lexing with flex
Previous:From: Tom LaneDate: 2004-02-18 00:43:21
Subject: Re: Doing psql's lexing with flex

Privacy Policy | About PostgreSQL
Copyright © 1996-2017 The PostgreSQL Global Development Group