Skip site navigation (1) Skip section navigation (2)

Re: [patch] libpq one-row-at-a-time API

From: Jan Wieck <JanWieck(at)Yahoo(dot)com>
To: Leon Smith <leon(dot)p(dot)smith(at)gmail(dot)com>
Cc: Merlin Moncure <mmoncure(at)gmail(dot)com>, Marko Kreen <markokr(at)gmail(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Postgres Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [patch] libpq one-row-at-a-time API
Date: 2012-07-31 03:26:59
Message-ID: 50175083.2060109@Yahoo.com (view raw or flat)
Thread:
Lists: pgsql-hackers
On 7/30/2012 10:31 PM, Leon Smith wrote:
> This is not necessarily true,  on multiple levels.   I mean,  some of
> the programs I write are highly concurrent,  and this form of batching
> would have almost no risk of stalling the network buffer.    And
> the possible use case would be when you are dealing with very small
> rows,  when there would typically be several rows inside a single
> network packet or network buffer.

With "highly concurrent" you mean multi-threaded? Like one thread reads 
the rows in batches and pushes them into a queue while another thread 
processes them from that queue?

If that is the case, then you just added a useless layer of buffering 
and the need for thread/thread context switches to PQsetSingleRowMode. 
Libpq's "receiver thread" is the kernel itself. Libpq tries to never 
read partial kernel buffers already. It always makes sure that there are 
at least 8K of free space in the inBuffer. In the case you describe 
above, where several rows fit into a single packet, libpq will receive 
them with a single system call in one read(2), then the application can 
get them as fast as possible, without causing any further context 
switches because they are already in the inBuffer.

I've written that sort of code myself in the past. Look at the Slony 
worker thread prior to 2.2. We switched to the COPY protocol instead of 
waiting for the single row mode and got rid of all that extra buffering 
already (and then some more).


Jan

-- 
Anyone who trades liberty for security deserves neither
liberty nor security. -- Benjamin Franklin

In response to

Responses

pgsql-hackers by date

Next:From: Qi HuangDate: 2012-07-31 05:50:14
Subject: [patch] pg_stat_lwlocks view - lwlocks statistics
Previous:From: Leon SmithDate: 2012-07-31 02:31:44
Subject: Re: [patch] libpq one-row-at-a-time API

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group