Re: [patch] libpq one-row-at-a-time API

On 7/30/2012 10:31 PM, Leon Smith wrote:
> This is not necessarily true, on multiple levels. I mean, some of
> the programs I write are highly concurrent, and this form of batching
> would have almost no risk of stalling the network buffer. And
> the possible use case would be when you are dealing with very small
> rows, when there would typically be several rows inside a single
> network packet or network buffer.

With "highly concurrent" you mean multi-threaded? Like one thread reads
the rows in batches and pushes them into a queue while another thread
processes them from that queue?

If that is the case, then you just added a useless layer of buffering
and the need for thread/thread context switches to PQsetSingleRowMode.
Libpq's "receiver thread" is the kernel itself. Libpq tries to never
read partial kernel buffers already. It always makes sure that there are
at least 8K of free space in the inBuffer. In the case you describe
above, where several rows fit into a single packet, libpq will receive
them with a single system call in one read(2), then the application can
get them as fast as possible, without causing any further context
switches because they are already in the inBuffer.

I've written that sort of code myself in the past. Look at the Slony
worker thread prior to 2.2. We switched to the COPY protocol instead of
waiting for the single row mode and got rid of all that extra buffering
already (and then some more).

Jan

--
Anyone who trades liberty for security deserves neither
liberty nor security. -- Benjamin Franklin