Re: [PATCHES] Post-special page storage TDE support

Hi Aleksander et al,

Enclosing v4 for this patch series, rebased atop the
constant-splitting series[1]. For the purposes of having cfbot happy,
I am including the prerequisites as a squashed commit v4-0000, however
this is not technically part of this series.

The roadmap this time is similar to the last series, with some
improvements being made in terms of a few bug fixes and other
reorganizations/cleanups. With the prerequisite/rework, we are able
to eliminate some number of patches in the previous series.

Squashed prerequisites, out of scope for review:
v4-0000-squashed-prerequisites.patch

Refactoring some of the existing uses of BLCKSZ and SizeOfPageHeaderData:
v4-0001-refactor-Create-PageUsableSpace-to-represent-spac.patch
v4-0002-refactor-Make-PageGetUsablePageSize-routine.patch
v4-0003-feature-Add-ReservedPageSize-variable.patch
v4-0004-feature-Adjust-page-sizes-at-PageInit.patch

Making TOAST dynamic:
v4-0005-feature-Add-hook-for-setting-reloptions-defaults-.patch
v4-0006-feature-Add-Calc-options-for-toast-related-pieces.patch
v4-0007-feature-Dynamically-calculate-toast_tuple_target.patch
v4-0008-chore-Replace-TOAST_MAX_CHUNK_SIZE-with-ClusterTo.patch
v4-0009-chore-Translation-updates-for-TOAST_MAX_CHUNK_SIZ.patch

Infra/support for blocksize calculations:
v4-0010-chore-Split-nbtree.h-structure-defs-into-an-inter.patch
v4-0011-Control-File-support-for-reserved_page_size.patch
v4-0012-feature-Calculate-all-blocksize-constants-in-a-co.patch

GUC/initdb/bootstrap support for setting reserved-page-size:
v4-0013-GUC-for-reserved_page_size.patch
v4-0014-feature-Add-reserved_page_size-to-initdb-bootstra.patch
v4-0015-feature-Updates-for-pg_resetwal.patch

Optimization of VisMap:
v4-0016-optimization-Add-support-for-fast-non-division-ba.patch
v4-0017-optimization-Use-fastdiv-code-in-visibility-map.patch

Docs:
v4-0018-doc-update-bufpage-docs-w-reserved-space-data.patch

Misc cleanup/fixes:
v4-0019-feature-Teach-bloom-about-PageUsableSpace.patch
v4-0020-feature-teach-FSM-about-reserved-page-space.patch
v4-0021-feature-expose-reserved_page_size-in-SQL-controld.patch

Write out of init options that are relevant:
v4-0022-feature-save-out-our-initialization-options.patch

A few notes:
- There was a bug in the previous VisMap in v3 which resulted in
treating the page size as smaller than it was. This has been fixed.

- v4-0022 is new, but useful for the page features going forward, and
should simplify some things like using `pg_resetwal` or other places
that really need to know how initdb was initialized.

- I have done some performance metrics with this feature vs unpatched
postgres. Since the biggest place this seemed to affect was the
visibility map (per profiling), I constructed an index-only scan test
case which basically measured nested loop against index-only lookups
with something like 20M rows in the index and 1M generate_series
options, measuring the differences between the approach we are using
(and several others), and showing a trimmean of < 0.005 in execution
time.[2] This seems acceptable (if not just noise), so would be
interested in any sorts of performance deviations others encounter.

Thanks,

David

[1] https://commitfest.postgresql.org/47/4828/
[2] https://www.pgguru.net/2024-03-13-vismap-benchmarking.txt