Re: pl/pgSQL, get diagnostics and big data

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: "Andreas 'ads' Scherbaum" <adsmail(at)wars-nicht(dot)de>
Cc: Christian Ullrich <chris(at)chrullrich(dot)net>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: pl/pgSQL, get diagnostics and big data
Date: 2016-03-12 03:30:12
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

I wrote:
> I'll take it from here, unless I find bigger issues.

Hmm ... so the more I pulled on this string, the more stuff I found.
The attached updated patch fixes several additional significant

* call_cntr and max_calls in FuncCallContext are now uint64

* Result widths for PortalRunFetch and allied routines are now uint64

* count arguments to ExecutorRun and related routines are now uint64

* Maximum number of tuples in a SPITupleTable is now uint64

* Management of portal (cursor) positions fixed. I promoted portalPos to
uint64, and got rid of posOverflow, which seems not especially necessary
if we're using 64-bit counters.

* A whole lot of places that were comparing integer-width variables
to SPI_processed, and would therefore go into infinite loops with
result sizes above 2G tuples, now use uint64 counters instead.

However, I drew the line at changing FetchStmt.howMany, which means that
the count-limit inputs to PortalRunFetch et al didn't change; they're
still signed long. Changing that would require messing with the lexer
which I did not feel like doing. So the maximum offset/position in a
FETCH or MOVE is (still) platform-dependent. I'm not sure that this is
worth changing, as you will get a clean syntax error if you try to exceed
the limit.

BTW, if anyone thinks the above changes represent too much API churn,
speak up. This is definitely going to cause some pain for extensions.

There's a fair amount of work yet to do:

1. I found two places (marked XXX in this patch) that are using strtoul()
to parse a tuple count back out of a command tag. That won't do anymore.
pg_stat_statements has a messy hack for the same problem (look for
HAVE_STRTOULL), which is probably what we want to do, but not by
copy-and-pasting #ifdef HAVE_STRTOULL into multiple places. I'd be
inclined to provide a utility function "pg_strtouint64" or some such
to encapsulate that. (numutils.c might be a good place for it.)

2. As I was just complaining to -hackers, plpython plperl and pltcl
all now contain attempts to pass uint64 values (from SPI_processed)
into language-specific functions. We need to figure out whether
that will overflow and whether it's worth doing something about.

3. This patch still needs a lot of review as I may have missed

So I'm bouncing this back to Waiting on Author.

regards, tom lane

Attachment Content-Type Size
64bit_5.patch text/x-diff 64.1 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message David Rowley 2016-03-12 03:31:33 Re: Parallel Aggregate
Previous Message Andres Freund 2016-03-12 03:15:02 Re: pgsql: Only try to push down foreign joins if the user mapping OIDs mat