Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi!

Michael and Hannu, here's a POC patch with direct TIDs TOAST.
The simplest implementation where we store a chain of TIDs, each
chunk stores the next TID to be fetched. Patch applies on top of
commit 998b0b51d5ea763be081804434f177082ba6772b (origin/toast_64bit_v2)
Author: Michael Paquier <michael(at)paquier(dot)xyz>
Date: Thu Jun 19 13:09:11 2025 +0900

While it is very fast on small data - I see several disadvantages:
- first of all, VACUUM should be revised to work with such tables;
- problematic batch insertion due to necessity to store TID chain.

It is just a POC implementation, so please don't blame me for
questionable decisions.

Any opinions and feedback welcome!

PS: Hannu, just seen your latest message, will check it out now.

On Mon, Jul 21, 2025 at 3:15 AM Hannu Krosing <hannuk(at)google(dot)com> wrote:

> I have been evolving details for Direct TOAST design in
> https://wiki.postgresql.org/wiki/DirectTOAST
>
> The top level goals are
>
> * 8-byte TOAST pointer - just (header:1, tag:1 and TID:6)
> * all other info moved from toast pointer to actual toast record(s),
> so heap rows are smaller and faster.
> * all extra fields are bytea with internal encoding (maybe will create
> full new types for these, or maybe just introspection functions are
> enough)
> the reasons for this are
> - PostgresSQL arrays add 20 byte overhead
> - bytea gives other freedoms in encoding for minimal space usage
>
> No solution yet for va_toastrelid , but hope is
> - to use some kind of mapping and find one or two free bits somewhere
> (tid has one free),
> - or add a 12-byte toast pointer just for this.
> - or to make sure that CLUSTER and VACUUM FULL can be done without
> needing va_toastrelid. I assume it is there for clustering the TOAST
> which will be not possible separately from the main heap with direct
> toast tid pointers anyway.
>
> Please take a look and poke holes in it !
>
>
>

--
Regards,
Nikita Malakhov
Postgres Professional
The Russian Postgres Company
https://postgrespro.ru/