Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Tue, Sep 16, 2025 at 02:14:43PM +0900, Michael Paquier wrote:
> On Thu, Aug 14, 2025 at 02:49:06PM +0900, Michael Paquier wrote:
>> I have dropped the amcheck test patch for now, which was fun but it's
>> not really necessary for the "basics". I have done also more tests,
>> playing for example with pg_resetwal, installcheck and pg_upgrade
>> scenarios. I am wondering if it would be worth doing a pg_resetwal in
>> the node doing an installcheck on the instance to be upgraded, bumping
>> its next OID to be much larger than 4 billion, actually..
>
> Four patches had conflicts with 748caa9dcb68, so rebased as v6.

There were a few conflicts, so here is a rebased v7, moving the patch
to the next CF. I have been sitting on this patch for six weeks for
the moment.

Tom, you are registered as a reviewer of the patch. The point of
contention of the patch, where I see there is no consensus yet, is if
my approach of using a redirection for the external TOAST pointers
with a new layer to facilitate the addition of more vartags (aka the
64b value vartag proposed here, concept that could also apply to
compression methods later on) is acceptable. Moving to a different
approach, like the "brutal" one I am naming upthread where the
redirection layer is replaced by changes in all the code paths that
need to be touched, would be of course cheaper at runtime as there
would be no more redirection, but the maintenance would be a nightmare
the more vartags we add, and I have some plans for more of these.
Doing the switch would be a few hours work, so that would not be a big
deal, I guess. The important part is an agreement about the approach,
IMO.

Please note that the latest patch set also uses a new reloption to
control if 8 byte TOAST values are set (not a GUC), binary upgrades
are handled with binary_upgrade.h functions, and the 8-byte OID value
uses a dedicated data type, as reviewed upthread.

One thing that shows up in the last patch of the set is mentioned in
an XXX comment in toast_external_assign_vartag(), if it would be
better to use a 4-byte vartag even if we have a 8-byte value in the
TOAST table to make the TOAST pointers shorter when a value is less
than 4 billions. Not sure how much to do about this one, and there's
little point in doing this change without the earlier infrastructure
patches if the approach taken is thought as OK, as well.
--
Michael