error in libpq reading large fields

Stephen Oberski (sfo(at)deterministic(dot)com) reports a bug with a severity of 1
The lower the number the more severe it is.

Short Description
error in libpq reading large fields

Long Description
Problem observed with Postgres 7.1.12 on SunOS 5.6.

Application using libpq running the following:

DECLARE companies CURSOR FOR SELECT c.streamableobject FROM company c, company_ric cr WHERE c.ric = cr.ric and cr.companyric = 'CM.TO'
FETCH ALL IN companies

would fail when c.streamableobject exceeded 16K bytes in size,
c.streamableobject being a TEXT field.

The problem seems to be in the libpq/fe-misc.c pqReadData() function.
Its heuristic for increasing the buffer size and re-reading does not work in all cases.

When the heuristic was replaced with code that continued to reallocate the buffer and reread as long as data was read on the previous read, the problem was corrected.

Note that this query worked fine in psql for any size of streamableobject.

Sample Code
Here is the diff of the modified fe-misc.c vrs. the original:

$ diff fe-misc.c ~/postgresql-7.1.2-/src/interfaces/libpq/fe-misc.c
420d419
< tryAgainWithRealloc:
471,473c470,480
< * Keep reading until the data dries up.
< * Need to increase the read buffer size if we get too
< * close to the end.
---
> * Hack to deal with the fact that some kernels will only give u
s
> * back 1 packet per recv() call, even if we asked for more and
> * there is more available. If it looks like we are reading a
> * long message, loop back to recv() again immediately, until we
> * run out of data or buffer space. Without this, the
> * block-and-restart behavior of libpq's higher levels leads to
> * O(N^2) performance on long messages.
> *
> * Since we left-justified the data above, conn->inEnd gives the
> * amount of data already read in the current message. We
> * consider the message "long" once we have acquired 32k ...
475,477c482,487
< someread = 1;
< goto tryAgainWithRealloc;
<
---
> if (conn->inEnd > 32768 &&
> (conn->inBufSize - conn->inEnd) >= 8192)
> {
> someread = 1;
> goto tryAgain;
> }
508d517
< * @@@ But what if conn->inBufSize - conn->inEnd == 0 ?

No file was uploaded with this report