Scanner performance (was Re: 7.3 schedule)

Tom Lane writes:

> We do have some numbers suggesting that the per-character loop in the
> lexer is slow enough to be a problem with very long literals. That is
> the overhead that might be avoided with a special protocol.

Which loop is that? Doesn't the scanner use buffered input anyway?

> However, it should be noted that (AFAIK) no one has spent any effort at
> all on trying to make the lexer go faster. There is quite a bit of
> material in the flex documentation about performance considerations ---
> someone should take a look at it and see if we can get any wins by being
> smarter, without having to introduce protocol changes.

My profiles show that the work spent in the scanner is really minuscule
compared to everything else.

The data appears to support a suspicion that I've had many moons ago that
the binary search for the key words takes quite a bit of time:

0.22 0.06 66748/66748 yylex [125]
[129] 0.4 0.22 0.06 66748 base_yylex [129]
0.01 0.02 9191/9191 yy_get_next_buffer [495]
0.02 0.00 32808/34053 ScanKeywordLookup [579]
0.00 0.01 16130/77100 MemoryContextStrdup [370]
0.00 0.00 4000/4000 scanstr [1057]
0.00 0.00 4637/4637 yy_get_previous_state [2158]
0.00 0.00 4554/4554 base_yyrestart [2162]
0.00 0.00 4554/4554 yywrap [2163]
0.00 0.00 1/1 base_yy_create_buffer [2852]
0.00 0.00 1/13695 base_yy_load_buffer_state [2107]

I while ago I've experimented with hash functions for the key word lookup
and got a speedup of factor 2.5, but again, this is really minor in the
overall scheme of things.

(The profile data is from a run of all the regression test files in order
in one session.)

--
Peter Eisentraut peter_e(at)gmx(dot)net