Subject: [RFC] Umbra: Reducing Full-Page Write Amplification through Physical Page Remapping

Hi hackers,

This email is probably a little long.

There have been previous discussions in the community about the cost of
PostgreSQL full_page_writes (FPW). Umbra is an exploration that continues
from that line of discussion.

Umbra does not attempt to disable FPW or change the page, buffer, or WAL
semantics at the PostgreSQL upper layers. Instead, it asks a more specific
question:

If a checkpoint has already persisted a reliable physical page, why does
the first modification after the checkpoint still need to write the entire
page as an FPI into WAL?

The core idea of Umbra is:

Keep the old physical page corresponding to the checkpoint as a recovery
baseline, and write post-checkpoint modifications to a new physical page
instead of overwriting that recovery baseline.

The following describes the design.

1. Why does in-place update require FPW?

With PostgreSQL's in-place page update model, if new data after a checkpoint
directly overwrites the physical page that represents the checkpoint
baseline, a crash may leave that physical page in a state that cannot be
trusted as a recovery starting point.

Therefore, with full_page_writes=on, the first modification of a page after
a checkpoint needs to write the complete page into WAL.

Conceptually:

checkpoint
|
v
page becomes recovery baseline
|
v
first modification after checkpoint
|
v
original physical page is overwritten
|
v
crash
|
v
original physical page may no longer be a reliable baseline
|
v
FPI is required

In this sense, an FPI stores a complete page-level recovery baseline in WAL.

Umbra asks:

If the checkpoint has already persisted this recovery baseline, can we
simply preserve that physical page instead of writing the same information
to WAL again?

2. The basic idea of Umbra

The core idea is that the physical page persisted by the checkpoint is not
overwritten by subsequent updates.

In traditional PostgreSQL, a logical page is essentially associated with
a fixed physical location.

Umbra decouples logical page identity from physical persistence:

logical page
|
v
physical mapping
|
+---+---+
| |
v v
source target

The source is the old physical page protected by the checkpoint.

The target is the new physical page used for modifications after the
checkpoint.

Post-checkpoint updates are written to the target while the source remains
unchanged.

If a crash occurs, recovery can start from the already durable source page
as the checkpoint recovery baseline and then apply the subsequent WAL.

In this way, the recovery baseline that would normally be provided by an FPI
in WAL is instead preserved as part of the physical page lifecycle.

3. Why are multiple physical generations required?

The key issue is that a checkpoint is not an instantaneous event.

Suppose the beginning and end of checkpoints are represented by c and e:

c1 ---------------- e1
checkpoint 1

c2 ---------------- e2
checkpoint 2

The database continues to generate updates while a checkpoint is in
progress.

Therefore, it is not sufficient to think of:

checkpoint = one instantaneous point

Instead, checkpoint overlap has to be considered.

To illustrate why multiple physical generations are required, suppose p1,
p2, and p3 represent the physical recovery baselines of the same logical
page in adjacent checkpoint generations:

c2 p1 e1 c3 p2 e2 c4 p3 e3

When p3 is produced, it is not yet known whether e3 has completed.

If a crash occurs before e3:

checkpoint 3 has not completed, so checkpoint 3 cannot yet be treated as
the effective recovery point.

Recovery may therefore still need the earlier physical recovery baseline:

p1

If e3 has already completed:

the newer generation has become effective, and recovery can use:

p2

Therefore, when p3 is produced, p1 cannot immediately be reclaimed.

This relationship continues as checkpoint generations advance.

The current prototype therefore keeps three adjacent physical generations:

previous-previous
previous
current

Three generations are the simple implementation used by the current
prototype to cover the different crash states introduced by overlapping
checkpoints.

From the physical layout perspective, suppose three consecutive logical
pages were originally stored as:

a b c

The current prototype may organize their physical generations as:

a1 a2 a3 b1 b2 b3 c1 c2 c3

where:

a1 a2 a3

are the three physical generations of logical page a,

b1 b2 b3

are the three physical generations of logical page b,

and similarly for c.

It is important to emphasize that three times the space is not a theoretical
requirement of the Umbra model. It is the space-amplification bound chosen
by the current prototype.

The three-generation implementation is primarily intended to reduce
implementation complexity and provide a direct way to validate the
correctness relationship between checkpoints, recovery, and physical
generations.

If more aggressive generation reclamation can be proven not to break the
recovery baseline at any crash point, the space amplification can be
reduced further.

4. Why physical remapping instead of COW?

A relatively direct implementation would be copy-on-write (COW).

The old page would remain unchanged and the new modification would be
written to a new buffer.

The COW model is easy to understand because it provides a very clear
boundary between the page state before and after the checkpoint.

However, there is a different observation here.

Because WAL replay is idempotent, the physical page that needs to be
protected by a checkpoint does not necessarily require a strict,
instantaneous page boundary similar to COW.

In other words, what recovery needs is a reliable recovery baseline. It does
not necessarily require copying the entire buffer at a precisely defined
point in time.

If such a copy is not required, then there is no reason to pay the additional
page-copy cost on the normal update path.

Therefore, Umbra chooses to decouple logical page identity from physical
persistence at the smgr layer.

The logical page identity remains unchanged, while the physical page
associated with it can change across checkpoint generations.

A mapping/selector state records which physical page currently represents
the logical page.

5. How is correctness guaranteed?

Umbra calls the operation that switches a logical page from an old physical
page to a new physical page a "shift".

5.1 Shift and WAL must remain consistent

The WAL record that produces a shift and the corresponding physical mapping
change must have consistent recovery visibility.

We must not have a state where:

the mapping has already switched to target

but:

the WAL record that describes this mapping change

is not yet visible to recovery.

Therefore, the shift and its corresponding WAL record cannot be split by
the checkpoint RedoRecPtr.

This prevents recovery from observing an intermediate state that could not
have existed during normal execution.

5.2 Checkpoint must preserve the old physical page

A checkpoint is a process rather than an instantaneous event.

During checkpoint scanning, a page may have already undergone a shift:

checkpoint starts
|
v
scan page
|
v
page is shifted
|
v
active physical page = target

However, the recovery baseline required by the checkpoint may still be:

source

Therefore, checkpoint processing cannot simply use the currently active
physical page to determine which physical page must be persisted.

It must be able to identify the old physical page required by the checkpoint
and ensure that this page becomes durable before the checkpoint completes.

In other words:

when the checkpoint completes, the corresponding recovery baseline must
already have been flushed and synced.

5.3 AIO must capture the physical page

Another issue is asynchronous I/O.

Suppose an AIO operation starts while:

logical page
|
v
physical page A

If a shift occurs afterwards, and the AIO operation looks up the physical
page again through the logical mapping, it could incorrectly issue an I/O
operation against B instead of A.

Therefore, Umbra records the physical page corresponding to the logical
page when the AIO operation starts.

An I/O operation that has already started will therefore continue to operate
on the physical block captured at its start, regardless of subsequent shifts.

6. What does Umbra reduce?

Umbra primarily targets:

WAL amplification caused by Full-Page Writes.

The traditional path is:

checkpoint
|
v
page modification
|
v
first modification after checkpoint
|
v
FPI
|
v
complete page written to WAL

Umbra changes this to:

checkpoint
|
v
source physical page
|
+--------------> recovery baseline
|
v
new modification
|
v
target physical page

The complete page state that would otherwise need to be stored in WAL can
instead be provided by the physical page preserved by the checkpoint.

Umbra does not claim to eliminate all forms of physical write amplification.

The primary target is the WAL amplification caused by FPIs.

7. Initial experimental results

The following results use the same TPC-C workload:

800 warehouses
400 clients
warmup: 2 min
run: 20 min
checkpoint: 4 min
completion_target: 0.9

PostgreSQL with full_page_writes=on:

tpmC: 254,422.92
WAL / NewOrder: 38,318.65 B

PostgreSQL with full_page_writes=off:

tpmC: 529,109.08
WAL / NewOrder: 8,056.66 B

Umbra with full_page_writes=on:

tpmC: 557,536.23
WAL / NewOrder: 8,002.68 B

The full_page_writes=off result is used only as a sensitivity baseline;
it is not a correctness baseline.

The important observation is the WAL amplification:

FPW ON

38,318.65 B / NewOrder
|
v
Umbra

8,002.68 B / NewOrder

which is already close to:

FPW OFF

8,056.66 B / NewOrder

At the same time, the throughput changes from:

254,422.92 tpmC

to:

557,536.23 tpmC

while the full_page_writes=off baseline is:

529,109.08 tpmC

These initial results suggest that, while keeping
full_page_writes=on, physical remapping can preserve the checkpoint
recovery baseline and substantially reduce the WAL amplification caused
by FPIs, bringing both WAL consumption and performance into the range of
the full_page_writes=off baseline.

8. Costs of the current prototype

The current implementation still has several visible costs.

8.1 Read amplification

Because the current prototype keeps multiple physical generations for a
logical page, logical pages that were previously physically contiguous may
map to different physical locations.

For example:

a b c

may become:

a1 a2 a3 b1 b2 b3 c1 c2 c3

This can break the original physical locality and therefore increase read
amplification.

However, this is primarily a physical-layout issue rather than an inherent
requirement of the physical-remapping model itself.

The physical placement and generation layout can be further optimized to
improve locality.

8.2 Space amplification

The current prototype uses three physical generations and therefore
introduces space amplification.

However, space amplification and write amplification are different costs.

The three generations are primarily used to cover the state space created
by checkpoint overlap and crash recovery. They do not mean that every
logical update needs to be written three times.

The current three-generation layout is primarily an implementation choice
to simplify the correctness argument and prototype implementation, rather
than a final physical-layout requirement.

If more aggressive generation reclamation can be proven not to break the
recovery baseline at any crash point, the space amplification can be
reduced further.

8.3 Metadata overhead

Physical remapping requires mapping/selector state and therefore introduces
additional metadata management overhead.

The current prototype primarily focuses on validating whether this
physical persistence model can correctly integrate with PostgreSQL's
existing WAL, checkpoint, buffer I/O, and recovery framework.

9. Summary

Umbra attempts to decouple logical page identity from physical persistence.

The physical page preserved by the checkpoint becomes the recovery
baseline, while post-checkpoint modifications are written to a target
physical page.

In this way, the page-level recovery baseline that would traditionally be
provided by an FPI in WAL can instead be provided by the physical page
lifecycle.

The current TPC-C experiment shows that, with full_page_writes=on, Umbra
reduces WAL/NewOrder from 38,318.65 bytes to 8,002.68 bytes, which is
already close to the 8,056.66 bytes observed with full_page_writes=off.

At the same time, throughput moves from 254K tpmC with FPW enabled to
557K tpmC with Umbra, compared with 529K tpmC for the full_page_writes=off
baseline.

The initial results therefore support the following hypothesis:

If a checkpoint already provides a reliable physical recovery baseline,
then it may not be necessary to store another complete copy of that page
in WAL through an FPI.

Umbra attempts to preserve this baseline through physical remapping and
to decouple logical page identity from physical persistence.

The main purpose of the current patch series is not to propose a final
physical layout, but to validate whether this persistence model can be
implemented correctly within PostgreSQL's existing WAL, checkpoint, buffer
I/O, and recovery framework, while reducing the WAL amplification caused
by FPW.

The follow-up patches are still under review. This email is intended to provide context for the design and facilitate further discussion.

Comments and discussion are very welcome.