[RFC PATCH v3 0/7] Umbra: a remap-aware smgr prototype

Hi hackers,

This is a follow-up to my previous Umbra RFC thread:

https://www.postgresql.org/message-id/flat/a8d4e49b-7f20-435f-8555-7907554fedaf%40nayishan.top

It has been about two months since my last message.

During that time, I did two main things. First, I refactored Umbra into a
three-slot physical placement model that is better suited to PostgreSQL.
Second, I optimized the implementation around that model. With
full_page_writes=on, Umbra's performance is now close to that of md with
full_page_writes=off.

These results were worth sharing promptly. I have completed a separate audit
of P1-P7, so I am sending that review unit now. My audit of the later series
is still in progress. Those patches already exist, but they are deliberately
not part of this review request.

The attached `umbra-p1-p7-20260808.tar.gz` contains P1-P7 only. Each patch
has its own commit message describing its capability, ownership boundary,
invariant, dependency, and non-goals.

SHA256 (umbra-p1-p7-20260808.tar.gz):

cc1ca7345b3d2503626643a825f57247414d6e12a5e84a15330c907c0b52931f

The review plan is:

1. P1-P7: smgr selection, the md-compatible physical umfile layer, private
MAP metadata bootstrap and residency, compatibility limits, and the first
working mapped layout.

2. Selector-shift WAL/redo/checkpoint publication and correctness.

3. Performance and operational optimization.

The first round introduces the infrastructure and layout state required for
later selector shifts: smgr selection, umfile, private MAP metadata, the
initial mapped layout, and selector reading and caching. It does not yet
introduce selector-shift WAL, checkpoint-overlap handling, compaction, or
other later mechanisms.

Umbra keeps logical-to-physical placement below smgr. MAP is private,
relation-bound physical-placement metadata owned by Umbra. Table and index
AMs continue to operate on logical block numbers and do not interpret MAP
pages, physical slots, or selectors.

The model costs three physical slots per logical page. They retain the
old-page version used as the redo baseline and rotate through the three slots
in order. Although this increases physical page storage, the WAL reduction
lowers overall storage amplification.

Here are recent TPC-C results. The workload used 800 warehouses for 20 minutes
with checkpoint_timeout=4min and checkpoint_completion_target=0.9.

terminals md FPW=on md FPW=off Umbra FPW=on
-------------------------------------------------------
200 385616 614035 604807
400 279927 540942 568584
800 192126 420854 426757

WAL per New-Order was:

terminals md FPW=on md FPW=off Umbra FPW=on
-------------------------------------------------------
200 26.05 KiB 7.53 KiB 7.54 KiB
400 36.25 KiB 7.83 KiB 7.81 KiB
800 37.71 KiB 8.40 KiB 8.24 KiB

The test host was a Xeon Platinum 8173M at 2.00 GHz with 112 logical CPUs,
125 GiB RAM, a SATA DC500R SSD, and XFS. This is a single-host,
single-workload result. md with full_page_writes=off is an upper-bound and
sensitivity reference, not a correctness-equivalent configuration. I am also
attaching the detailed benchmark summary and hardware report.

For testing or for reading the complete design beyond this first review scope,
the currently relatively stable branch is:

https://github.com/nayishan/postgre_umbra/tree/chunksplit-restack

That branch is intended for running tests and understanding the complete
design. The submitted series will remain split into the review rounds
described above.

The complete work is still substantial. If anything in P1-P7 or its background
is unclear, please contact me and I will provide more context.

Regards,
Mingwei Jia