Re: [POC] Fast COPY FROM command for the table with foreign partitions

From: "Andrey V(dot) Lepikhov" <a(dot)lepikhov(at)postgrespro(dot)ru>
To: Amit Langote <amitlangote09(at)gmail(dot)com>, Alexey Kondratov <a(dot)kondratov(at)postgrespro(dot)ru>
Cc: Michael Paquier <michael(at)paquier(dot)xyz>, Ashutosh Bapat <ashutosh(dot)bapat(at)2ndquadrant(dot)com>, Ashutosh Bapat <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com>, Etsuro Fujita <etsuro(dot)fujita(at)gmail(dot)com>, PostgreSQL-Dev <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [POC] Fast COPY FROM command for the table with foreign partitions
Date: 2020-09-10 09:57:43
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On 9/9/20 5:51 PM, Amit Langote wrote:
> On Wed, Sep 9, 2020 at 6:42 PM Alexey Kondratov
> <a(dot)kondratov(at)postgrespro(dot)ru> wrote:
>> On 2020-09-09 11:45, Andrey V. Lepikhov wrote:
>>> This does not seem very convenient and will lead to errors in the
>>> future. So, I agree with Amit.
>> And InitResultRelInfo() may set ri_usesMultiInsert to false by default,
>> since it's used only by COPY now. Then you won't need this in several
>> places:
>> + resultRelInfo->ri_usesMultiInsert = false;
>> While the logic of turning multi-insert on with all the validations
>> required could be factored out of InitResultRelInfo() to a separate
>> routine.
> Interesting idea. Maybe better to have a separate routine like Alexey says.
Ok. I rewrited the patch 0001 with the Alexey suggestion.
Patch 0002... required minor changes (new version see in attachment).

Also I added some optimization (see 0003 and 0004 patches). Here we
execute 'COPY .. FROM STDIN' at foreign server only once, in the
BeginForeignCopy routine. It is a proof-of-concept patches.

Also I see that error messages processing needs to be rewritten. Unlike
the INSERT operation applied to each row, here we find out copy errors
only after sending the END of copy. Currently implementations 0002 and
0004 provide uninformative error messages for some cases.

Andrey Lepikhov
Postgres Professional

Attachment Content-Type Size
v9-0001-Move-multi-insert-decision-logic-into-executor.patch text/x-patch 16.5 KB
v9-0002-Fast-COPY-FROM-into-the-foreign-or-sharded-table.patch text/x-patch 35.8 KB
v9-0003-Add-separated-connections-into-the-postgres_fdw.patch text/x-patch 9.6 KB
v9-0004-Optimized-version-of-the-Fast-COPY-FROM-feature.patch text/x-patch 10.9 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Kyotaro Horiguchi 2020-09-10 09:58:59 Re: v13: CLUSTER segv with wal_level=minimal and parallel index creation
Previous Message Surafel Temesgen 2020-09-10 09:55:04 Re: Evaluate expression at planning time for two more cases