Re: BUG #16722: PG hanging on COPY when table has close to 2^32 toasts in the table.

Andres Freund <andres(at)anarazel(dot)de> writes:
> On 2020-11-18 19:02:34 -0500, Tom Lane wrote:
>> Yeah. If we're going to put work into this, widening the IDs used
>> to identify toast values seems like the right work to be doing.

> To outline, here's what I think the two major pieces to get there are:

> 1) Make toast oid assignment independent of the oid counter. The easiest
> way likely is to also create a serial and use that. That alone
> improves the situation considerably, because it takes much longer to
> to wrap around in each toast table.

I'm not really on board with creating an actual sequence object for each
toast table. I don't think they are efficient enough.

I had imagined internally widening the OID counter to 64 bits, of which
ordinary OID usage would just discard the high 32 bits, but for TOAST
chunk IDs we could use the whole value. Now, compared to your idea
that would result in using the wider toast-pointer format more often,
but I think there are ways around that. For example, we could do

(1) Grab the next 64-bit OID.
(2) Probe the toast table to see if the low 32 bits correspond to
a value in use in that toast table. If not, just use the low 32 bits.
(3) Otherwise, use the whole 64 bits and the wider pointer format.
(We could do another uniqueness probe here, but I think it's unnecessary,
since the 64-bit counter will never wrap around.)

Since the existing OID assignment mechanism requires a uniqueness
probe already, this isn't any more expensive than what we do now.

regards, tom lane