Re: Direct TOAST v2, faster, smaller and no migration needed

Rebased

Conflict resolved in Commit 0001 (detoast.c): Upstream commit
e27f3b2cad7 ("Remove duplicate definition of the compressed varlena
header") replaced TOAST_COMPRESS_METHOD / TOAST_COMPRESS_EXTSIZE with
VARDATA_COMPRESSED_GET_COMPRESS_METHOD /
VARDATA_COMPRESSED_GET_EXTSIZE. Resolved in
toast_decompress_datum_slice() to use
VARDATA_COMPRESSED_GET_COMPRESS_METHOD(attr).

On Fri, Sep 25, 2026 at 8:49 AM Hannu Krosing <hannuk(at)google(dot)com> wrote:
>
> [PATCH v10 0/9] Direct TOAST: bypass TOAST B-Tree index via physical TIDs
> ============================================================
>
> Hello hackers,
>
> Here is version 10 of the Direct TOAST patch set.
>
> Direct TOAST introduces an alternative out-of-line varlena storage format
> that bypasses the traditional TOAST B-Tree index scans during reads and index
> insertions during writes by embedding physical TIDs directly in the varlena
> pointer (varatt_direct) and within hierarchical chunk trees.
>
> This update addresses several concurrency, resource safety, and permission
> invariants identified during recent reviews, and adds interrupt checks and
> recursion guards across chunk traversal paths.
>
> The updated branch is also pushed to:
> https://github.com/postsql/postgres/tree/direct-toast-submission
>
>
> Key Changes in v10
> ==================
>
> 1. Superuser Privilege Check for In-Place Upgrades
> - Added an explicit superuser check (superuser()) in pg_ensure_direct_toast()
> before modifying catalog state. Only superusers may alter TOAST table
> metadata and mark TOAST indexes partial.
> - Added regression test coverage in direct_toast.sql / direct_toast.out
> verifying that unprivileged roles receive ERRCODE_INSUFFICIENT_PRIVILEGE.
>
> 2. Relation Locking and Transaction Lifetime Invariants
> - In ensure_direct_toast(), upgraded the target relation lock from
> AccessShareLock to ShareUpdateExclusiveLock while catalog columns and
> index predicates are being modified, preventing concurrent conflicting
> schema modifications.
> - Followed standard backend conventions by using table_close(..., NoLock) on
> both the target relation and toast relation in ensure_direct_toast(),
> ensuring locks are held until transaction commit rather than released
> prematurely.
> - In toast_delete_datum_direct(), changed table_close(toastrel,
> RowExclusiveLock)
> to table_close(toastrel, NoLock) to avoid premature lock release/downgrade
> before the transaction finishes.
>
> 3. Safe Buffer Pin Cleanup on Corrupted Chunk Error Paths
> - In toast_fetch_datum_slice() (detoast.c), explicitly call
> ReleaseBuffer(buffer)
> before invoking elog(ERROR, ...) on unexpected NULL chunk data or corrupted
> chunk headers, preventing buffer pin leaks during error recovery.
>
> 4. Recursion Depth Protection (Stack Safety)
> - Added check_stack_depth() calls to all recursive chunk tree traversals:
> * toast_fetch_datum_direct_slice_recursive() in detoast.c
> * toast_delete_datum_direct_recursive() in toast_internals.c
> * check_toasted_attribute_direct_recursive() in
> contrib/amcheck/verify_heapam.c
>
> 5. Interrupt Responsiveness in Iteration Loops
> - Added CHECK_FOR_INTERRUPTS() in chunk iteration loops across the backend
> to ensure statement timeouts and cancel requests (SIGINT) are processed
> promptly when processing very large TOAST DAGs:
> * Multi-chunk tree slice traversal in detoast.c
> * Direct chunk deletion loop in toast_internals.c
> * Logical decoding chunk reconstruction in reorderbuffer.c
> * Recursive child verification loop in contrib/amcheck/verify_heapam.c
>
> 6. Pointer Size Documentation and Static Assertions
> - Updated varatt.h and README.toast to document that sizeof(varatt_direct)
> is 20 bytes (matching varatt_external_oid8 due to 2 bytes of trailing
> compiler struct alignment padding), resulting in a 22-byte on-disk
> external varlena header.
> - Added a compile-time StaticAssertDecl on sizeof(varatt_direct) to ensure
> struct layout invariants are enforced across platforms.
>
>
> Patch Series Overview (v10)
> ===========================
>
> v10-0001: Refactor detoasting and decompression pipeline to unify full and
> slice operations.
> v10-0002: Add Direct TOAST catalog, GUC, and reloptions infrastructure.
> v10-0003: Implement Direct TOAST core storage reading and writing (flat array
> and hierarchical tree DAG, recursive slicing, deletion).
> v10-0004: Support Direct TOAST in logical decoding, replication, and online
> REPACK.
> v10-0005: Add amcheck verification for Direct TOAST tuples (verify_heapam).
> v10-0006: Add documentation for Direct TOAST.
> v10-0007: Add pg_ensure_direct_toast for in-place legacy TOAST table upgrade.
> v10-0008: Add backend TOAST architecture documentation and clean up detoast
> access.
> v10-0009: Prune dead unindexed TOAST tuples directly to LP_UNUSED
> during VACUUM.
>
>
> Testing & Verification
> ======================
>
> The patch series builds cleanly under GCC and Clang with:
> CFLAGS="-O2 -ggdb -fsanitize=alignment,undefined -fno-sanitize-recover=all"
> CPPFLAGS="-DRELCACHE_FORCE_RELEASE
> -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS"
> --enable-cassert --enable-injection-points --enable-tap-tests
>
> Verification results:
> - make check in src/test/regress: 240/240 tests passed (including
> direct_toast).
> - make check in contrib/amcheck: 4 regression tests and 6 TAP suites (298/298
> tests) passed.
> - make check in src/test/modules/injection_points: 4 regression tests and
> 15 isolation tests (including repack_direct_toast) passed.
> - Server log audit confirmed zero buffer refcount leaks, zero cache leaks,
> and zero resource warnings under RELCACHE_FORCE_RELEASE.
>
> Feedback and review are warmly welcome.
>
> Best regards,
> Hannu