Re: POC: postgres_fdw insert batching

On 11/25/20 7:31 AM, tsunakawa(dot)takay(at)fujitsu(dot)com wrote:
> From: Craig Ringer <craig(dot)ringer(at)enterprisedb(dot)com>
>> I suggest that when developing this, you keep in mind the ongoing
>> work on the libpq pipelining/batching enhancements, and also the
>> way many interfaces to foreign data sources support asynchronous,
>> concurrent operations.
>
> Yes, thank you, I bear it in mind. I understand it's a feature for
> batching multiple kinds of SQL statements like DBC's batch updates.
>

I haven't followed the libpq pipelining thread very closely. It does
seem related, but I'm not sure if it's a good match for this patch, or
how far is it from being committable ...

>
>> I'd argue it's pretty much vital for decent performance when
>> talking to a cloud database from an on-prem server for example, or
>> any other time that round-trip-time reduction is important.
>
> Yeah, I'm thinking of the data migration and integration as the
> prominent use case.
>

Well, good that we all agree this is a useful feature to have (in
general). The question is whether postgres_fdw should be doing batching
on it's onw (per this thread) or rely on some other feature (libpq
pipelining). I haven't followed the other thread, so I don't have an
opinion on that.

Note however we're doing two things here, actually - we're implementing
custom batching for postgres_fdw, but we're also extending the FDW API
to allow other implementations do the same thing. And most of them won't
be able to rely on the connection library providing that, I believe.

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company