From: | Merlin Moncure <mmoncure(at)gmail(dot)com> |
---|---|
To: | Jan Wieck <JanWieck(at)yahoo(dot)com> |
Cc: | Leon Smith <leon(dot)p(dot)smith(at)gmail(dot)com>, Marko Kreen <markokr(at)gmail(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Postgres Hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: [patch] libpq one-row-at-a-time API |
Date: | 2012-07-31 14:12:57 |
Message-ID: | CAHyXU0xDHoF_U1nKA9b+i5O7p_XV0jEj4yAARGrhz0YR3uiAXw@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Mon, Jul 30, 2012 at 10:26 PM, Jan Wieck <JanWieck(at)yahoo(dot)com> wrote:
> On 7/30/2012 10:31 PM, Leon Smith wrote:
>>
>> This is not necessarily true, on multiple levels. I mean, some of
>> the programs I write are highly concurrent, and this form of batching
>> would have almost no risk of stalling the network buffer. And
>> the possible use case would be when you are dealing with very small
>> rows, when there would typically be several rows inside a single
>> network packet or network buffer.
>
>
> With "highly concurrent" you mean multi-threaded? Like one thread reads the
> rows in batches and pushes them into a queue while another thread processes
> them from that queue?
>
> If that is the case, then you just added a useless layer of buffering and
> the need for thread/thread context switches to PQsetSingleRowMode. Libpq's
> "receiver thread" is the kernel itself. Libpq tries to never read partial
> kernel buffers already. It always makes sure that there are at least 8K of
> free space in the inBuffer. In the case you describe above, where several
> rows fit into a single packet, libpq will receive them with a single system
> call in one read(2), then the application can get them as fast as possible,
> without causing any further context switches because they are already in the
> inBuffer.
Yeah: with asynchronous query processing the query gets sent and
control returns immediately to your code: that's the whole point.
Even if some data races to the network buffer, libpq doesn't 'see' any
data until you tell it to by asking for a result (which can block) or
draining the buffers with PQconsumeInput. So there is no race in the
traditional sense and I'm ok with the PQsetSingleRowMode as such.
Removing malloc/free on row iteration seems only to be possible via
one of two methods: either a) you introduce a non-PGresult based
method of data extraction or b) you preserve the PGresult across row
iterations.
merlin
From | Date | Subject | |
---|---|---|---|
Next Message | Adam Mackler | 2012-07-31 14:47:20 | Re: Learning SQL: nested CTE and UNION |
Previous Message | Jonatan Reiners | 2012-07-31 13:57:40 | Re: Learning SQL: nested CTE and UNION |