Re: [HACKERS] Lex and things...

Ansley, Michael wrote:
>
> Hi,
>
> Shot, Leon. The patch removes the #define YY_USES_REJECT from scan.c, which
> means we now have expandable tokens. Of course, it also removes the
> scanning of "embedded minuses", which apparently causes the optimizer to
> unoptimize a little.

Oh, no. Unary minus gets to grammar parser and there is recognized as
such. Then for numeric constants it becomes an *embedded* minus in
function doNegate. So unary minus after parser in numeric constants
is embedded minus, as it was earlier before patch. In other words,
I can see no change in representation of grammar after patching.

> However, the next step is attacking the limit on the
> size of string literals. These seemed to be wired to YY_BUF_SIZE, or
> something. Is there any reason for this?

Hmm. There is something going on to remove fixed length limits
entirely, maybe someone is already doing something to lexer in
that respect? If no, I could look at what can be done there.

--
Leon.