Re: postgres_fdw: Use COPY to speed up batch inserts

From: Kirill Reshke <reshkekirill(at)gmail(dot)com>
To: Matheus Alcantara <matheusssilv97(at)gmail(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>, Andrew Dunstan <andrew(at)dunslane(dot)net>, jian he <jian(dot)universality(at)gmail(dot)com>, Tomas Vondra <tomas(at)vondra(dot)me>, solaimurugan vellaipandiyan <drsolaimurugan(dot)v(at)gmail(dot)com>
Subject: Re: postgres_fdw: Use COPY to speed up batch inserts
Date: 2026-09-09 20:16:16
Message-ID: CALdSSPhi0ca0zEz3qTBYeCg-J=R72uc4dZF-_0XDf9U3q1RDPw@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Thu, 10 Sept 2026 at 01:10, I <reshkekirill(at)gmail(dot)com> wrote:

>

>
> By the way I noticed that batch_size is practically always no more
> than 1000 in batch insert callback (even though configured to a bigger
> value in relation options.). I didn't look for the exact reason.
>

I did actually take a fast look:

```
/*
* Send the buffered tuples to the FDW in batches of at most
* batch_size, as we do for ExecForeignBatchInsert. Each call is a
* self-contained COPY operation.
*
* COPY provides no RETURNING, so this path is only usable when
* there are no AFTER ROW triggers that would need the stored rows.
*/
while (sent < nused)
{
int size = (batch_size < nused - sent) ? batch_size : (nused - sent);

resultRelInfo->ri_FdwRoutine->ExecForeignBatchCopy(estate,
resultRelInfo,
&slots[sent],
size);

sent += size;

/* Update the row counter and progress of the COPY command */
*processed += size;
pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED,
*processed);
}
```

In this place, big batch_size (1e5-1e6) is always capped with 1000 by
CopyMultiInsertBuffer logic. At least for me this is the case. Maybe
this is another place for adjusting in sight of this feature.

--
Best regards,
Kirill Reshke

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Bharath Rupireddy 2026-09-09 20:45:42 Re: Optimize UUID parse using SIMD
Previous Message Andres Freund 2026-09-09 20:12:30 Re: pg_get_*_ddl() needs a redesign