Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Mon, Jul 07, 2025 at 05:33:11PM +0300, Nikita Malakhov wrote:
> [VARTAG_ONDISK_OID] = {
> .toast_pointer_size = TOAST_POINTER_INT8_SIZE, <--- here
> .maximum_chunk_size = TOAST_MAX_CHUNK_SIZE_OID,
> .to_external_data = ondisk_oid_to_external_data,
> .create_external_data = ondisk_oid_create_external_data,
> },
> };
>
> Shouldn't TOAST_POINTER_INT8_SIZE be replaced with TOAST_POINTER_OID_SIZE?

Yes, thanks for pointing this out. This one has lurked in one of the
rebases (not sure how) and it was impacting the threshold calculation
where we consider if an attribute should be compressed or not. I have
taken this occasion to work a bit more on the patch set. The patch
structure is mostly the same, with two tweaks because I was unhappy
with these in the initial patch set:
- The addition of a new callback able to retrieve a new TOAST value,
to ease the diffs in toast_save_datum().
- Reordering of the patch set, with the TOAST external refactoring
done much earlier in the series, now placed in 0003.

The most interesting piece of the patch is still 0003 "Refactor
external TOAST pointer code for better pluggability". On top of that
stands a 0004 patch named "Introduce new callback to get fresh TOAST
values" where I have added value conflict handling for int8. The
split makes reviews easier, hopefully.

Please note that I still need to look at perf profiles and some flame
graphs with the refactoring done in 0003 with the worst case I've
mentioned upthread with detoasting and values stored uncompressed in
the TOAST relation.

I have also pushed this v2 on this branch, so feel free to grab it if
that makes your life easier:
https://github.com/michaelpq/postgres/tree/toast_64bit_v2
--
Michael