Re: Emitting JSON to file using COPY TO

From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: jian he <jian(dot)universality(at)gmail(dot)com>
Cc: Joe Conway <mail(at)joeconway(dot)com>, Junwang Zhao <zhjwpku(at)gmail(dot)com>, Florents Tselai <florents(dot)tselai(at)gmail(dot)com>, "Andrey M(dot) Borodin" <x4mmm(at)yandex-team(dot)ru>, Dean Rasheed <dean(dot)a(dot)rasheed(at)gmail(dot)com>, Daniel Verite <daniel(at)manitou-mail(dot)org>, Davin Shearer <davin(at)apache(dot)org>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Emitting JSON to file using COPY TO
Date: 2026-03-06 12:33:31
Message-ID: 57db4b6e-51ba-4984-9343-bc62a3ee0b7c@dunslane.net
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-general pgsql-hackers


On 2026-03-06 Fr 4:38 AM, jian he wrote:
> COPY (SELECT 1 UNION ALL SELECT 2) TO stdout WITH (format json);
>
> still fails with v25-0002, json_tupledesc_ready is not helpful.
> I think I figured it out. We need to use BlessTupleDesc in BeginCopyTo.
> Then let slot->tts_tupleDescriptor point to cstate->queryDesc->tupDesc
> in CopyToJsonOneRow
>
> * CSV, text and json formats share the same TextLike routines except for the
> * one-row callback.
> This comment is not useful, I want to delete it.
>
> CopyToTextLikeStart
> + /* JSON-specific initialization */
> + if (cstate->opts.format == COPY_FORMAT_JSON)
> + {
> + MemoryContext oldcxt;
> +
> + /* Allocate reusable JSON output buffer in long-lived context */
> + oldcxt = MemoryContextSwitchTo(cstate->copycontext);
> + initStringInfo(&cstate->json_buf);
> + MemoryContextSwitchTo(oldcxt);
> + }
> We ca just add
>
> cstate->json_buf = makeStringInfo();
>
> in BeginCopyTo.
>
> v25-0004-COPY-TO-JSON-build-JSON-per-column-support-colum.patch
> added several fields to the CopyToStateData.
> Actually, there is a simpler way (construct a new Tupdesc and let
> composite_to_json do the job), please see my v26-0004.
>

Yeah, I got some crashes using this, but reworked it a bit, and all now
seems good. I was apparently mistaken about the supposed speedup, but
the performance is steady.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Attachment Content-Type Size
v27-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch text/x-patch 13.1 KB
v27-0002-json-format-for-COPY-TO.patch text/x-patch 23.3 KB
v27-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch text/x-patch 12.3 KB
v27-0004-COPY-TO-JSON-support-column-lists.patch text/x-patch 12.5 KB

In response to

Browse pgsql-general by date

  From Date Subject
Next Message Adrian Klaver 2026-03-06 16:03:07 Re: Index (primary key) corrupt?
Previous Message Wim Rouquart 2026-03-06 10:31:04 RE: Index (primary key) corrupt?

Browse pgsql-hackers by date

  From Date Subject
Next Message Andrei Lepikhov 2026-03-06 12:55:23 Re: Skipping schema changes in publication
Previous Message Jonathan Gonzalez V. 2026-03-06 12:10:12 Re: pg_upgrade fails when extension_control_path is used