Re: [HACKERS] Query length string

On Wed, Aug 11, 1999 at 02:42:55PM +0200, Ansley, Michael wrote:
> I have gotten as far as the tcop, now I need some help as far as
> yacc(bison?) goes. Yacc seems to use an input buffer called YY_BUFFER or
> something, which appears to be a fixed length string. Does anybody have any
> idea how I go about changing this so that I can pass a char * which I have
> allocated (apart from reading the manual, I'm looking for the 'technical
> summary' ;-)?

I presume your talking about the YY_BUF_SIZE mentioned in
backend/parse/Makefile? Earlier in this list, Thomas Lockart mentioned that
we're really flex specific, not general lex (dupport for an exclusive
start state, I think he said.) Tom Lane alludes to that in his reply as well.
Given that, this excerpt from the flex docs is useful:

--------------
Note that yytext can be defined in two different ways: either as
a character pointer or as a character array. You can control which
definition flex uses by including one of the special directives `%pointer'
or `%array' in the first (definitions) section of your flex input. The
default is `%pointer', unless you use the `-l' lex compatibility option,
in which case yytext will be an array. The advantage of using `%pointer'
is substantially faster scanning and no buffer overflow when matching very
^^^^^^^^^^^^^^^^^^
large tokens (unless you run out of dynamic memory). The disadvantage is
that you are restricted in how your actions can modify yytext (see the
next section), and calls to the `unput()' function destroys the present
contents of yytext, which can be a considerable porting headache when
moving between different lex versions.

The advantage of `%array' is that you can then modify yytext to your
heart's content, and calls to `unput()' do not destroy yytext (see
below). Furthermore, existing lex programs sometimes access yytext
externally using declarations of the form:

extern char yytext[];

This definition is erroneous when used with `%pointer', but correct for
`%array'.

`%array' defines yytext to be an array of YYLMAX characters, which
defaults to a fairly large value. You can change the size by simply
#define'ing YYLMAX to a different value in the first section of your
flex input. As mentioned above, with `%pointer' yytext grows dynamically
^^^^^^^^^^^^^^^^^
to accommodate large tokens. While this means your `%pointer' scanner
^^^^^^^^^^^^^^^^^^^^^^^^^^^
can accommodate very large tokens (such as matching entire blocks of
comments), bear in mind that each time the scanner must resize yytext it
also must rescan the entire token from the beginning, so matching such
tokens can prove slow. yytext presently does not dynamically grow if a
call to `unput()' results in too much text being pushed back; instead,
a run-time error results.

Also note that you cannot use `%array' with C++ scanner classes (the c++
option; see below).

------------

I've checked, and the parser generated from our parse.l does in fact do
the dynamic buffer resize bit, so the token buffer (char *) passed on to
the grammar generated by yacc(bison) is already of variable size. However,
it looks like there are hand coded maximum SQL query length checks in
the parser there you'll need to zap.

HTH,
Ross

--
Ross J. Reedstrom, Ph.D., <reedstrm(at)rice(dot)edu>
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St., Houston, TX 77005