Re: Direct TOAST v2, faster, smaller and no migration needed

[PATCH v10 0/9] Direct TOAST: bypass TOAST B-Tree index via physical TIDs
============================================================

Hello hackers,

Here is version 10 of the Direct TOAST patch set.

Direct TOAST introduces an alternative out-of-line varlena storage format
that bypasses the traditional TOAST B-Tree index scans during reads and index
insertions during writes by embedding physical TIDs directly in the varlena
pointer (varatt_direct) and within hierarchical chunk trees.

This update addresses several concurrency, resource safety, and permission
invariants identified during recent reviews, and adds interrupt checks and
recursion guards across chunk traversal paths.

The updated branch is also pushed to:
https://github.com/postsql/postgres/tree/direct-toast-submission

Key Changes in v10
==================

1. Superuser Privilege Check for In-Place Upgrades
- Added an explicit superuser check (superuser()) in pg_ensure_direct_toast()
before modifying catalog state. Only superusers may alter TOAST table
metadata and mark TOAST indexes partial.
- Added regression test coverage in direct_toast.sql / direct_toast.out
verifying that unprivileged roles receive ERRCODE_INSUFFICIENT_PRIVILEGE.

2. Relation Locking and Transaction Lifetime Invariants
- In ensure_direct_toast(), upgraded the target relation lock from
AccessShareLock to ShareUpdateExclusiveLock while catalog columns and
index predicates are being modified, preventing concurrent conflicting
schema modifications.
- Followed standard backend conventions by using table_close(..., NoLock) on
both the target relation and toast relation in ensure_direct_toast(),
ensuring locks are held until transaction commit rather than released
prematurely.
- In toast_delete_datum_direct(), changed table_close(toastrel,
RowExclusiveLock)
to table_close(toastrel, NoLock) to avoid premature lock release/downgrade
before the transaction finishes.

3. Safe Buffer Pin Cleanup on Corrupted Chunk Error Paths
- In toast_fetch_datum_slice() (detoast.c), explicitly call
ReleaseBuffer(buffer)
before invoking elog(ERROR, ...) on unexpected NULL chunk data or corrupted
chunk headers, preventing buffer pin leaks during error recovery.

4. Recursion Depth Protection (Stack Safety)
- Added check_stack_depth() calls to all recursive chunk tree traversals:
* toast_fetch_datum_direct_slice_recursive() in detoast.c
* toast_delete_datum_direct_recursive() in toast_internals.c
* check_toasted_attribute_direct_recursive() in
contrib/amcheck/verify_heapam.c

5. Interrupt Responsiveness in Iteration Loops
- Added CHECK_FOR_INTERRUPTS() in chunk iteration loops across the backend
to ensure statement timeouts and cancel requests (SIGINT) are processed
promptly when processing very large TOAST DAGs:
* Multi-chunk tree slice traversal in detoast.c
* Direct chunk deletion loop in toast_internals.c
* Logical decoding chunk reconstruction in reorderbuffer.c
* Recursive child verification loop in contrib/amcheck/verify_heapam.c

6. Pointer Size Documentation and Static Assertions
- Updated varatt.h and README.toast to document that sizeof(varatt_direct)
is 20 bytes (matching varatt_external_oid8 due to 2 bytes of trailing
compiler struct alignment padding), resulting in a 22-byte on-disk
external varlena header.
- Added a compile-time StaticAssertDecl on sizeof(varatt_direct) to ensure
struct layout invariants are enforced across platforms.

Patch Series Overview (v10)
===========================

v10-0001: Refactor detoasting and decompression pipeline to unify full and
slice operations.
v10-0002: Add Direct TOAST catalog, GUC, and reloptions infrastructure.
v10-0003: Implement Direct TOAST core storage reading and writing (flat array
and hierarchical tree DAG, recursive slicing, deletion).
v10-0004: Support Direct TOAST in logical decoding, replication, and online
REPACK.
v10-0005: Add amcheck verification for Direct TOAST tuples (verify_heapam).
v10-0006: Add documentation for Direct TOAST.
v10-0007: Add pg_ensure_direct_toast for in-place legacy TOAST table upgrade.
v10-0008: Add backend TOAST architecture documentation and clean up detoast
access.
v10-0009: Prune dead unindexed TOAST tuples directly to LP_UNUSED
during VACUUM.

Testing & Verification
======================

The patch series builds cleanly under GCC and Clang with:
CFLAGS="-O2 -ggdb -fsanitize=alignment,undefined -fno-sanitize-recover=all"
CPPFLAGS="-DRELCACHE_FORCE_RELEASE
-DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS"
--enable-cassert --enable-injection-points --enable-tap-tests

Verification results:
- make check in src/test/regress: 240/240 tests passed (including
direct_toast).
- make check in contrib/amcheck: 4 regression tests and 6 TAP suites (298/298
tests) passed.
- make check in src/test/modules/injection_points: 4 regression tests and
15 isolation tests (including repack_direct_toast) passed.
- Server log audit confirmed zero buffer refcount leaks, zero cache leaks,
and zero resource warnings under RELCACHE_FORCE_RELEASE.

Feedback and review are warmly welcome.

Best regards,
Hannu