Re: RFI: Extending the TOAST Pointer

On Thu, 18 May 2023 at 15:12, Aleksander Alekseev
<aleksander(at)timescale(dot)com> wrote:
>
> Hi,
>
> > I agree that va_tag can have another use. But since we are going to
> > make varatt_external variable in size (otherwise I don't see how it
> > could be really **extendable**) I don't think this is the right
> > approach.

Why would we modify va_tag=18; data=varatt_external? A different
va_tag option would allow us to keep the current layout around without
much maintenance and a consistent low overhead.

> On second thought, perhaps we are talking more or less about the same thing?
>
> It doesn't matter what will be used as a sign of presence of a varint
> bitmask in the pointer. My initial proposal to use ToastCompressionId
> for this is probably redundant since va_tag > 18 will already tell
> that.

I'm not sure "extendable" would be the right word, but as I see it:

1. We need more space to store more metadata;
2. Essentially all bits in varatt_external are already accounted for; and
3. There are still many options left in va_tag

It seems to me that adding a new variant to va_att for marking new
features in the toast infrastructure makes the most sense, as we'd
also be able to do the new things like varints etc without needing to
modify existing toast paths significantly.

We'd need to stop using the va_tag as length indicator, but I don't
think it's currently assumed to be a length indicator anyway (see
VARSIZE_EXTERNAL(ptr)). By not using the varatt_external struct
currently in use, we could be able to get down to <18B toast pointers
as well, though I'd consider that unlikely.

Kind regards,

Matthias van de Meent
Neon, Inc.