Re: libpq Alternate Row Processor

From: Kyle Gearhart <kyle(dot)gearhart(at)indigohill(dot)io>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: libpq Alternate Row Processor
Date: 2017-02-08 23:11:20
Message-ID: BLUPR14MB0162071912F5872BA84757DBFA420@BLUPR14MB0162.namprd14.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]:
>> Kyle Gearhart <kyle(dot)gearhart(at)indigohill(dot)io> writes:
>>> The guts of pqRowProcessor in libpq does a good bit of work to maintain the internal data structure of a PGresult. There are a few use cases where the caller doesn't need the ability to access the result set row by row, column by column using PQgetvalue. Think of an ORM that is just going to copy the data from PGresult for each row into its own structures.

>> It seems like you're sort of reinventing "single row mode":
https://www.postgresql.org/docs/devel/static/libpq-single-row-mode.html

>> Do we really need yet another way of breaking the unitary-query-result abstraction?

> If it's four times faster...then the option should be available in libpq. I'm traveling tomorrow but will try to get a patch and proof with pgbench dataset up by the middle of the week.

Attached is a proof, test program and test results. No documentation changes have been included at this time.

It was tested against a pgbench_accounts record set with 100,000 records. Overall, wall clock improves 24%. User time elapsed is a 430% improvement. About half the time is spent waiting on the IO with the callback. With the regular pqRowProcessor only about 16% of the time is spent waiting on IO.

The test program follows the pgbench program's command line options, with an added parameter called "m", short for mode. Set the option to "row" for single row processing and "cb" for callback processing.

I did not provision for the test program to accept a password from a prompt, you'll have to pass that in the arguments.

Attachment Content-Type Size
libpq_rp.patch application/octet-stream 5.8 KB
performance.zip.zip application/x-zip-compressed 10.2 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tom Lane 2017-02-08 23:54:43 Re: Improve OR conditions on joined columns.
Previous Message Jim Nasby 2017-02-08 22:31:26 Re: Press Release Draft - 2016-02-09 Cumulative Update