Re: PATCH: Batch/pipelining support for libpq

From: "Daniel Verite" <daniel(at)manitou-mail(dot)org>
To: "Andres Freund" <andres(at)anarazel(dot)de>
Cc: "Craig Ringer" <craig(at)2ndquadrant(dot)com>,"Vaishnavi Prabakaran" <vaishnaviprabakaran(at)gmail(dot)com>,"Michael Paquier" <michael(dot)paquier(at)gmail(dot)com>,"David Steele" <david(at)pgmasters(dot)net>,"Prabakaran, Vaishnavi" <VaishnaviP(at)fast(dot)au(dot)fujitsu(dot)com>,"Haribabu Kommi" <kommi(dot)haribabu(at)gmail(dot)com>,"Tsunakawa, Takayuki" <tsunakawa(dot)takay(at)jp(dot)fujitsu(dot)com>,"Dmitry Igrishin" <dmitigr(at)gmail(dot)com>,"PostgreSQL Hackers" <pgsql-hackers(at)postgresql(dot)org>,"Manuel Kniep" <m(dot)kniep(at)web(dot)de>,"fujita(dot)etsuro(at)lab(dot)ntt(dot)co(dot)jp" <fujita(dot)etsuro(at)lab(dot)ntt(dot)co(dot)jp>,"Iwata, Aya" <iwata(dot)aya(at)jp(dot)fujitsu(dot)com>
Subject: Re: PATCH: Batch/pipelining support for libpq
Date: 2017-06-22 11:43:35
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Andres Freund wrote:

- if (pqFlush(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /*
+ * Give the data a push. In nonblock mode, don't complain if
we're unable
+ * to send it all; PQgetResult() will do any additional
flushing needed.
+ */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ }

Seems to be responsible for roughly an 1.7x speedup in tps and equivalent
decrease in latency, based on the "progress" info.
I wonder how much of that corresponds to a decrease in the number of
packets versus the number of syscalls. Both matter, I guess.

But OTOH there are certainly batch workloads where it will be preferrable
for the first query to reach the server ASAP, rather than waiting to be
coalesced with the next ones.
libpq is not going to know what's best.
One option may be to leave that decision to the user by providing a
PQBatchAutoFlush(true|false) property, along with a PQBatchFlush()
function. Maybe we could even let the user set the size of the sending
buffer, so those who really want to squeeze performance may tune it
for their network and workload.

Best regards,
Daniel Vérité
PostgreSQL-powered mailer:
Twitter: @DanielVerite

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message amul sul 2017-06-22 11:48:28 Multi column range partition table
Previous Message Thomas Munro 2017-06-22 11:38:01 Re: Regarding Postgres Dynamic Shared Memory (DSA)