Re: Emitting JSON to file using COPY TO

From: jian he <jian(dot)universality(at)gmail(dot)com>
To: Andrew Dunstan <andrew(at)dunslane(dot)net>
Cc: Joe Conway <mail(at)joeconway(dot)com>, Junwang Zhao <zhjwpku(at)gmail(dot)com>, Florents Tselai <florents(dot)tselai(at)gmail(dot)com>, "Andrey M(dot) Borodin" <x4mmm(at)yandex-team(dot)ru>, Dean Rasheed <dean(dot)a(dot)rasheed(at)gmail(dot)com>, Daniel Verite <daniel(at)manitou-mail(dot)org>, Davin Shearer <davin(at)apache(dot)org>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Emitting JSON to file using COPY TO
Date: 2026-03-09 03:48:26
Message-ID: CACJufxGm3OhbG3=bj4nLxMzq92A84v6Q1dG+hR_ZFrfLg1B99w@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-general pgsql-hackers

On Mon, Mar 9, 2026 at 3:44 AM Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
> Hmm. But should we be scribbling on slot->tts_tupleDescriptor like that?
> How about something like this?:
>
> - * Full table or query without column list. Ensure the slot uses
> - * cstate->tupDesc so that the datum is stamped with the right type;
> - * for queries output type is RECORDOID this must be the blessed
> - * descriptor so that composite_to_json can look it up via
> - * lookup_rowtype_tupdesc.
> + * Full table or query without column list. For queries, the slot's
> + * TupleDesc may carry RECORDOID, which is not registered in the
> type
> + * cache and would cause composite_to_json's lookup_rowtype_tupdesc
> + * call to fail. Build a HeapTuple stamped with the blessed
> + * descriptor so the type can be looked up correctly.
> */
> if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid ==
> RECORDOID)
> - slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc;
> + {
> + HeapTuple tup;
>
> - rowdata = ExecFetchSlotHeapTupleDatum(slot);
> + tup = heap_form_tuple(cstate->tupDesc,
> + slot->tts_values,
> + slot->tts_isnull);
> + rowdata = HeapTupleGetDatum(tup);
> + }
> + else
> + {
> + rowdata = ExecFetchSlotHeapTupleDatum(slot);
> + }
>
This is better. I've tried to get rid of json_projvalues and json_projnulls.
Just using heap_form_tuple, but it won't work.

I incorporated the v28-0004 COPY column list into v9-0002.
With this patch set, we added four fields to the struct CopyToStateData.

+ StringInfo json_buf; /* reusable buffer for JSON output,
+ * initialized in BeginCopyTo */
+ TupleDesc tupDesc; /* Descriptor for JSON output; for a column
+ * list this is a projected descriptor */
+ Datum *json_projvalues; /* pre-allocated projection values, or
+ * NULL */
+ bool *json_projnulls; /* pre-allocated projection nulls, or NULL */

Using the script in
https://www.postgresql.org/message-id/CACJufxFFZqxC3p4WjpTEi4riaJm%3DpADX%2Bpy0yQ0%3DRWTn5cqK3Q%40mail.gmail.com
I tested it again on macOS and Linux, and there are no regressions for
COPY TO with the TEXT and CSV formats.

--
jian
https://www.enterprisedb.com/

Attachment Content-Type Size
v29-0002-json-format-for-COPY-TO.patch text/x-patch 29.3 KB
v29-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch text/x-patch 13.1 KB
v29-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch text/x-patch 12.3 KB

In response to

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Christoph Moench-Tegeder 2026-03-09 07:18:25 Re: CREATE TABLE fails
Previous Message Adrian Klaver 2026-03-09 01:22:12 Re: CREATE TABLE fails

Browse pgsql-hackers by date

  From Date Subject
Next Message Richard Guo 2026-03-09 04:01:14 Re: Convert NOT IN sublinks to anti-joins when safe
Previous Message Ashutosh Bapat 2026-03-09 03:47:21 Re: Options to control remote transactions’ access/deferrable modes in postgres_fdw