Re: [POC] Fast COPY FROM command for the table with foreign partitions

From: Andrey Lepikhov <a(dot)lepikhov(at)postgrespro(dot)ru>
To: Etsuro Fujita <etsuro(dot)fujita(at)gmail(dot)com>
Cc: PostgreSQL-Dev <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [POC] Fast COPY FROM command for the table with foreign partitions
Date: 2020-06-22 06:13:30
Message-ID: 993b6293-15fc-154c-e5b8-48d49d8bdc76@postgrespro.ru
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

19.06.2020 19:58, Etsuro Fujita пишет:
> On Tue, Jun 2, 2020 at 2:51 PM Andrey Lepikhov
> <a(dot)lepikhov(at)postgrespro(dot)ru> wrote:
>> Hiding the COPY code under the buffers management machinery allows us to
>> generalize buffers machinery, execute one COPY operation on each buffer
>> and simplify error handling.
>
> I'm not sure that it's really a good idea that the bulk-insert API is
> designed the way it's tightly coupled with the bulk-insert machinery
> in the core, because 1) some FDWs might want to send tuples provided
> by the core to the remote, one by one, without storing them in a
> buffer, or 2) some other FDWs might want to store the tuples in the
> buffer and send them in a lump as postgres_fdw in the proposed patch
> but might want to do so independently of MAX_BUFFERED_TUPLES and/or
> MAX_BUFFERED_BYTES defined in the bulk-insert machinery.
>
> I agree that we would need special handling for cases you mentioned
> above if we design this API based on something like the idea I
> proposed in that thread.
Agreed
>
>> As i understand, main idea of the thread, mentioned by you, is to add
>> "COPY FROM" support without changes in FDW API.
>
> I don't think so; I think we should introduce new API for this feature
> to keep the ExecForeignInsert() API simple.
Ok
>
>> All that I can offer in this place now is to introduce one new
>> ExecForeignBulkInsert(buf) routine that will execute single "COPY FROM
>> STDIN" operation, send tuples and close the operation. We can use the
>> ExecForeignInsert() routine for each buffer tuple if
>> ExecForeignBulkInsert() is not supported.
>
> Agreed.
In the next version (see attachment) of the patch i removed Begin/End
fdwapi routines. Now we have only the ExecForeignBulkInsert() routine.

--
Andrey Lepikhov
Postgres Professional

Attachment Content-Type Size
v3-0001-Fast-COPY-FROM-into-the-foreign-or-sharded-table.patch text/x-patch 23.3 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Dilip Kumar 2020-06-22 06:25:49 Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
Previous Message Bharath Rupireddy 2020-06-22 05:55:54 [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit