Re: Incremental results from libpq

Greg,

While you can bulk collect or array process records in Oracle,
the older less efficient method is also still around. And yes you can
actually measure the inefficiency thereby created. Also there are
places where that inefficiency is of value. Take a case where your
applying a price change to a pricing table, assume that you want to
raise the price by one of several values based on another column in the
table. Oracle allows you to open the cursor with the "for update of"
clause. This has the effect or locking the affected rows in update mode
as they are included in the result set. Now once you've figured out the
new price you can simply update the table "where current of
<cursor_name>". Works very well, especially in a stored
procedure/package/function where sqlnet does not get in the way.

-----Original Message-----
From: gsstark(at)mit(dot)edu [mailto:gsstark(at)mit(dot)edu]
Sent: Sunday, November 13, 2005 11:14 AM
To: Alvaro Herrera
Cc: Goulet, Dick; Frank van Vugt; Greg Stark;
pgsql-interfaces(at)postgresql(dot)org
Subject: Re: [INTERFACES] Incremental results from libpq

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:

> Postgres supports cursors too. The Qt guys, and everyone else, could
be
> using it to get incremental results right now, no libpq mods
necessary.

Not really since the way Postgres supports cursors is at the SQL level.
Users
of Qt and every other driver could be using cursors if their drivers
support
them, but Qt can't really be reasonably expected to go into users' SQL
and
modify them to use cursors.

Moreover cursors aren't really that great a substitute. With cursors you
have
to manually fetch individual records. You're just trading off the
inefficiencies of batching up all the results for entirely different
inefficiencies. Now for every record you retrieve you need a network
round
trip as well as a round trip down through your driver, the kernel layers
on
both machines, and the backend as well.

The efficient approach as Oracle and other mature network layers
implement is
to issue the query once, then pipeline the results back to the
application
buffering a substantial amount in the driver. DBD::Oracle goes to some
lengths
to ensure the number of records buffered is a reasonable multiple of the
default TCP mss of 1500 bytes.

So even though the application only retrieves one record at a time it's
just
pulling it out of an array that's already prefilled. When the array gets
low
the next block of records is retrieved from the server (where they're
probably
already buffered as well). The result is a constant flow of network
traffic
that keeps the application and network as busy as possible.

--
greg