Re: jsonpath

From: Nikita Glukhov <n(dot)gluhov(at)postgrespro(dot)ru>
To: Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com>
Cc: Michael Paquier <michael(at)paquier(dot)xyz>, Stas Kelvich <s(dot)kelvich(at)postgrespro(dot)ru>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>, Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com>, David Steele <david(at)pgmasters(dot)net>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Andrew Dunstan <andrew(dot)dunstan(at)2ndquadrant(dot)com>, Alexander Korotkov <a(dot)korotkov(at)postgrespro(dot)ru>, Robert Haas <robertmhaas(at)gmail(dot)com>, Oleg Bartunov <obartunov(at)gmail(dot)com>, Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
Subject: Re: jsonpath
Date: 2018-11-09 15:50:05
Views: Raw Message | Whole Thread | Download mbox
Lists: pgsql-hackers

Attached 20th version the jsonpath patches.

On 08.11.2018 4:52, Tomas Vondra wrote:

> On 11/6/18 4:48 PM, Tomas Vondra wrote:
>> On 11/6/18 3:31 PM, Nikita Glukhov wrote:
>>> On 29.10.2018 2:20, Tomas Vondra wrote:>
>>>> 9) It's generally a good idea to make the individual pieces committable
>>>> separately, but that means e.g. the regression tests have to pass after
>>>> each patch. At the moment that does not seem to be the case for 0002,
>>>> see the attached file. I'm running with -DRANDOMIZE_ALLOCATED_MEMORY,
>>>> not sure if that's related.
>>> This should definitely be a bug in json support, but I can't reproduce
>>> it simply by defining -DRANDOMIZE_ALLOCATED_MEMORY.  Could you provide
>>> a stack trace at least?
>> I'll try.
> Not sure why you can't reproduce the failures, it's perfectly
> reproducible for me. For the record, I'm doing this:
> ./configure --prefix=/home/user/pg-jsonpath --enable-debug
> --enable-cassert CFLAGS="-O0 -DRANDOMIZE_ALLOCATED_MEMORY" && make -s
> clean && make -s -j4 && make check
> After sticking Assert(false) to JsonEncodeJsonbValue (to the default
> case), I get a failure like this:
> select json '{}' @* 'lax $[0]';
> ! WARNING: unknown jsonb value type: 20938064
> ! server closed the connection unexpectedly
> ! This probably means the server terminated abnormally
> ! before or while processing the request.
> ! connection to server was lost
> The backtrace is attached. My guess is JsonValueListGetList in
> jsonb_jsonpath_query only does shallow copy instead of copying the
> pieces into funcctx->multi_call_memory_ctx, so it gets corrupted on
> subsequent calls.
> I also attach valgrind report, but I suppose the reported issues are a
> consequence of the same bug.

Than you for your help, I finally have managed to reproduce this bug.

The problem was really related to copying values but it was not located in the
jsonb_jsonpath_query() where the whole value list should already be allocated in
funcctx->multi_call_memory_ctx (executeJsonPath() is called in that context and
also input json/jsonb is copied into it). Stack-allocated root JsonbValue was
not copied to the heap due to wrong copy condition in recursiveExecuteNoUnwrap()
(jpiIndexArray case):

@@ -1754,1 +1754,1 @@ recursiveExecuteNoUnwrap(JsonPathExecContext *cxt, ...
- res = recursiveExecuteNext(cxt, jsp, &elem, v, found, !binary);
+ res = recursiveExecuteNext(cxt, jsp, &elem, v, found, singleton || !binary);

I have refactored a bit this place extracting explicit 'copy' flag.

Also I found a similar problem in makePassingVars() where Numerics were not
copied into 'multi_call_memory_ctx' from the input json object. I decided to use
datumCopy() inside makePassingVars() instead of copying the whole input json in
jsonb_jsonpath_query() using PG_GETARG_JSONB_P_COPY(). But I think that
processing of the input json with variables should ideally be bone lazily,
just don't know if this refactoring is worth it because 3rd argument of
json[b]_jsonpath_xxx() functions is used only for testing of variables passing,
SQL/JSON functions use the different mechanism.

On 06.11.2018 18:48, Tomas Vondra wrote:

>> But FF7-FF9 are questionable since the maximal supported precision is only 6.
>> They are optional by the standard:
>>    95) Specifications for Feature F555, “Enhanced seconds precision”:
>>      d) Subclause 9.44, “Datetime templates”:
>>        i) Without Feature F555, “Enhanced seconds precision”,
>>           a <datetime template fraction> shall not be FF7, FF8 or FF9.
>> So I decided to allow FF7-FF9 only on the output in to_char().
> Hmmmm, isn't that against the standard then? I mean, if our precision is
> only 6, that probably means we don't have the F555 feature, which means
> FF7-9 should not be available.
> It also seems like a bit surprising behavior that we only allow those
> fields in to_char() and not in other places.

Ok, now error is thrown if FF7-FF9 are found in the format string.

On 06.11.2018 18:48, Tomas Vondra wrote:

>>> 4) There probably should be .gitignore rule for jsonpath_gram.h, just
>>> like for other generated header files.
>> I see 3 rules in/src/backend/utils/adt/.gitignore:
>> /jsonpath_gram.h
>> /jsonpath_gram.c
>> /jsonpath_scan.c
> But there's a symlink in src/include/utils/jsonpath_gram.h and that's
> not in .gitignore.

Added jsonpath_gram.h to/src/include/utils/.gitignore.

Nikita Glukhov
Postgres Professional:
The Russian Postgres Company

Attachment Content-Type Size
0001-Preliminary-datetime-infrastructure-v20.patch text/x-patch 32.1 KB
0002-Jsonpath-engine-and-operators-v20.patch text/x-patch 343.8 KB
0003-Jsonpath-GIN-support-v20.patch text/x-patch 42.8 KB
0004-Jsonpath-syntax-extensions-v20.patch text/x-patch 46.4 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Adam Berlin 2018-11-09 15:51:23 Re: COPY FROM WHEN condition
Previous Message Dmitry Dolgov 2018-11-09 15:23:57 Re: Make description of heap records more talkative for flags