Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Tue, Jul 08, 2025 at 09:31:29PM +0300, Nikita Malakhov wrote:
> Michael, one more thing forgot to mention yesterday -
> #define TOAST_EXTERNAL_INFO_SIZE (VARTAG_ONDISK_OID + 1)
> static const toast_external_info
> toast_external_infos[TOAST_EXTERNAL_INFO_SIZE]
> VARTAG_ONDISK_OID historically has a value of 18
> and here we got an array of 19 members with only 2 valid ones.

Yeah, I'm aware of that. The code is mostly to make it easier to read
while dealing with this historical behavior, even if it costs a bit
more in memory. I don't think that it's a big deal, and we could
always have one more level of redirection to reduce its size. Now
there's the extra complexity..

> What do you think about having an individual
> TOAST value id counter per relation instead of using
> a common one? I think this is a very promising approach,
> but a decision must be made where it should be stored.

I've thought about that, and decided to discard this idea for now to
keep the whole proposal simpler. This has benefits if you have many
relations with few OIDs consumed, but this has a cost in itself as you
need to maintain the data for each TOAST relation. When I looked at
the problems a couple of weeks ago, I came to the conclusion that all
the checkbox properties of "local" TOAST values are filled with a
sequence: WAL logging to ensure uniqueness, etc. So I was even
considering the addition of some code to create sequences on-the-fly,
but at the end that was just more complexity with how we define
sequences currently compared to a unique 8-byte counter in the
control file that's good enough for a veeery long time.

I've also noticed that this sort of links to a piece I've implemented
last year and is still sitting in the CF app without much interest
from others: sequence AMs. You could implement a "TOAST" sequence
method, for example, optimized for this purpose. As a whole, I
propose to limit the scope of the proposal to the pluggability of the
external TOAST pointers. The approach I've taken should allow such
improvements, these can be added later if really needed.
--
Michael