Re: [PATCH] Infinite loop while acquiring new TOAST Oid

Andres Freund <andres(at)anarazel(dot)de> writes:
> On 2022-11-28 16:57:53 -0500, Tom Lane wrote:
>> As I said before, I think there's a decent argument that some people
>> will want the option to stay with 4-byte TOAST OIDs indefinitely,
>> at least for smaller tables.

> And as you said earlier, the increased overhead inside the toast table / index
> is not relevant compared to the size of toasted datums.

Perhaps not.

> I think we'll need to do something about the width of varatt_external to make
> the conversion to 64bit toast oids viable - and if we do, I don't think
> there's a decent argument for staying with 4 byte toast OIDs. I think the
> varatt_external equivalent would end up being smaller in just about all cases.

I agree that we can't simply widen varatt_external to use 8 bytes for
the toast ID in all cases. Also, I now get the point about avoiding
use of globally assigned OIDs here: if the counter starts from zero
for each table, then a variable-width varatt_external could actually
be smaller than currently for many cases. However, that bit is somewhat
orthogonal, and it's certainly not required for fixing the basic problem.

So it seems like the plan of attack ought to be:

1. Invent a new form or forms of varatt_external to allow different
widths of the toast ID. Use the narrowest width possible for any
given ID value.

2. Allow TOAST tables/indexes to store either 4-byte or 8-byte IDs.
(Conversion could be done as a side effect of table-rewrite
operations, perhaps.)

3. Localize ID selection so that tables can have small toast IDs
even when other tables have many IDs. (Optional, could do later.)

regards, tom lane