| From: | Kirill Reshke <reshkekirill(at)gmail(dot)com> |
|---|---|
| To: | Matheus Alcantara <matheusssilv97(at)gmail(dot)com> |
| Cc: | pgsql-hackers(at)postgresql(dot)org, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>, Andrew Dunstan <andrew(at)dunslane(dot)net>, jian he <jian(dot)universality(at)gmail(dot)com>, Tomas Vondra <tomas(at)vondra(dot)me>, solaimurugan vellaipandiyan <drsolaimurugan(dot)v(at)gmail(dot)com> |
| Subject: | Re: postgres_fdw: Use COPY to speed up batch inserts |
| Date: | 2026-09-09 20:16:16 |
| Message-ID: | CALdSSPhi0ca0zEz3qTBYeCg-J=R72uc4dZF-_0XDf9U3q1RDPw@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
On Thu, 10 Sept 2026 at 01:10, I <reshkekirill(at)gmail(dot)com> wrote:
>
>
> By the way I noticed that batch_size is practically always no more
> than 1000 in batch insert callback (even though configured to a bigger
> value in relation options.). I didn't look for the exact reason.
>
I did actually take a fast look:
```
/*
* Send the buffered tuples to the FDW in batches of at most
* batch_size, as we do for ExecForeignBatchInsert. Each call is a
* self-contained COPY operation.
*
* COPY provides no RETURNING, so this path is only usable when
* there are no AFTER ROW triggers that would need the stored rows.
*/
while (sent < nused)
{
int size = (batch_size < nused - sent) ? batch_size : (nused - sent);
resultRelInfo->ri_FdwRoutine->ExecForeignBatchCopy(estate,
resultRelInfo,
&slots[sent],
size);
sent += size;
/* Update the row counter and progress of the COPY command */
*processed += size;
pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED,
*processed);
}
```
In this place, big batch_size (1e5-1e6) is always capped with 1000 by
CopyMultiInsertBuffer logic. At least for me this is the case. Maybe
this is another place for adjusting in sight of this feature.
--
Best regards,
Kirill Reshke
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Bharath Rupireddy | 2026-09-09 20:45:42 | Re: Optimize UUID parse using SIMD |
| Previous Message | Andres Freund | 2026-09-09 20:12:30 | Re: pg_get_*_ddl() needs a redesign |