Re: WIP Incremental JSON Parser

On Mon, Mar 25, 2024 at 6:15 PM Jacob Champion <
jacob(dot)champion(at)enterprisedb(dot)com> wrote:

> On Wed, Mar 20, 2024 at 11:56 PM Andrew Dunstan <andrew(at)dunslane(dot)net>
> wrote:
> > Thanks, included that and attended to the other issues we discussed. I
> think this is pretty close now.
>
> Okay, looking over the thread, there are the following open items:
> - extend the incremental test in order to exercise the semantic callbacks
> [1]
>

Yeah, I'm on a super long plane trip later this week, so I might get it
done then :-)

> - add Assert calls in impossible error cases [2]
>

ok, will do

> - error out if the non-incremental lex doesn't consume the entire token [2]
>

ok, will do

> - double-check that out of memory is an appropriate failure mode for
> the frontend [3]
>

Well, what's the alternative? The current parser doesn't check stack depth
in frontend code. Presumably it too will eventually just run out of memory,
possibly rather sooner as the stack frames could be more expensive than
the incremental parser stack extensions.

>
> Just as a general style nit:
>
> > + if (lex->incremental)
> > + {
> > + lex->input = lex->token_terminator = lex->line_start = json;
> > + lex->input_length = len;
> > + lex->inc_state->is_last_chunk = is_last;
> > + }
> > + else
> > + return JSON_INVALID_LEXER_TYPE;
>
> I think flipping this around would probably make it more readable;
> something like:
>
> if (!lex->incremental)
> return JSON_INVALID_LEXER_TYPE;
>
> lex->input = ...
>
>
>
Noted. will do, Thanks.

cheers

andrew

> [1]
> https://www.postgresql.org/message-id/CAOYmi%2BnHV55Uhz%2Bo-HKq0GNiWn2L5gMcuyRQEz_fqpGY%3DpFxKA%40mail.gmail.com
> [2]
> https://www.postgresql.org/message-id/CAD5tBcLi2ffZkktV2qrsKSBykE-N8CiYgrfbv0vZ-F7%3DxLFeqw%40mail.gmail.com
> [3]
> https://www.postgresql.org/message-id/CAOYmi%2BnY%3DrF6dJCzaOuA3d-3FbwXCcecOs_S1NutexFA3dRXAw%40mail.gmail.com
>