Re: [patch] libpq one-row-at-a-time API

On Thu, Aug 2, 2012 at 8:19 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Marko Kreen <markokr(at)gmail(dot)com> writes:
>> On Wed, Aug 1, 2012 at 6:18 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> So I'm working from the first set of patches in your message
>>> <20120721194907(dot)GA28021(at)gmail(dot)com>.
>
>> Great!
>
> Here's an updated draft patch. I've not looked at the docs yet so
> this doesn't include that, but I'm reasonably happy with the code
> changes now. The main difference from what you had is that I pushed
> the creation of the single-row PGresults into pqRowProcessor, so that
> they're formed immediately while scanning the input message. That
> other method with postponing examination of the rowBuf does not work,
> any more than it did with the original patch, because you can't assume
> that the network input buffer is going to hold still. For example,
> calling PQconsumeInput after parseInput has absorbed a row-data message
> but before calling PQgetResult would likely break things.
>
> In principle I suppose we could hack PQconsumeInput enough that it
> didn't damage the row buffer while still meeting its API contract of
> clearing the read-ready condition on the socket; but it wouldn't be
> simple or cheap to do so.

Any code using single-row-mode is new. Any code calling consumeInput
before fully draining rows from buffer is buggy, as it allows unlimited growth
of network buffer, which breaks whole reason to use single-row mode.

It was indeed bug in previous patch that it did not handle this situation,
but IMHO it should be handled with error and not with allowing such code
to work.

Previously, whatever sequence the fetch functions were called, the apps
max memory usage was either 1x resultset size, or max 2x resultset size,
if it messed the sequence somehow. But no more. So it app knew
that resultset was big, it needed to split it up.

Now with single-row-mode, the apps can assume their max memory usage
is 1*row size + network buffer, lets simplify that to 2x row size.
But no more. And now they can process huge resultsets assuming
their memory usage will be no more than 2x row size.

And now libpq allows such apps to go from 2x row size to full resultset
size with unfortunate coding. Why?

I have opinions about reorg details too, but that's mostly matter of taste,
so rather unimportant compared to question whether we should allow
code to break the guarantees the single-row-mode gives.

--
marko