[RFC PATCH 0/19] Umbra: a remap-aware storage manager prototypeHi hackers,
This is an RFC for the current complete Umbra patch series.
Unlike the previous review round, this submission includes the complete
19-patch series.
The series contains 19 patches, organized into four parts:
Standalone fix: P1
Infrastructure: P2-P7
Core mechanism: P8-P14
Optimizations: P15-P19
Current branch:
https://github.com/nayishan/postgre_umbra/tree/restack/active-slot-three-buckets-no-root-fold-unlogged-map
Standalone fix: P1
P1: Fix repeated FPI generation when reusing hash overflow pages.
Infrastructure: P2-P7
P2: Add the build-time Umbra storage manager option and selection mechanism.
P3: Add an Umbra relation physical storage layer derived from md.
P4: Introduce a three-slot physical layout for permanent MAIN relations.
P5: Fix assumptions about stable physical block numbers in PostgreSQL utilities.
P6: Add the selector MAP cache and handle unlogged/temporary relations.
P7: Cache the active slot in shared buffers.
These patches establish the physical layout and logical-to-physical mapping infrastructure required by Umbra.
Core Mechanism: P8-P14
P8: Separate Umbra WAL record assembly from the md path.
P9: Implement MAIN slot shift using the source slot as the redo baseline.
P10: Reset the selector when a logical block is re-extended after truncate.
P11: Preserve the source slot during checkpoint so that recovery can access the baseline.
P12: Propagate the resolved physical block through AIO, write, writeback, and reopen paths.
P13: Extend the three-slot physical layout to permanent FSM and VM.
P14: Add active-slot and slot-shift redo handling for FSM and VM.
I consider P9 and P11 to be the two most important patches in the current series:
- P9 defines the core Umbra slot-shift mechanism.
- P11 addresses preservation of the recovery baseline across checkpoint overlap.
P12 primarily addresses correctness when physical remapping interacts with asynchronous I/O.
Optimizations: P15-P19
P15: Use byte-delta WAL records for mapped hint modifications.
P16: Add a background MAP writer.
P17: Avoid metadata I/O on selector cache hits.
P18: Merge writeback requests by physical block.
P19: Store Umbra state in BufferDesc.
These patches mainly reduce metadata, writeback, and cache-management overhead after the core mechanism is in place.
The core design and correctness model are described in my previous email. This email focuses mainly on the organization and implementation of the current patch series.
Initial Experimental Results
Test configuration:
800 warehouses
400 clients
warmup: 2 min
run: 20 min
checkpoint: 4 min
completion_target: 0.9
Key results:
tpmC WAL / NewOrder
FPW ON 254,422.92 38,318.65 B
FPW OFF 529,109.08 8,056.66 B
Umbra + FPW ON 557,536.23 8,002.68 B
With full_page_writes=on, Umbra brings WAL / NewOrder close to the full_page_writes=off baseline, while tpmC also reaches a similar range.
Current Status
This email is intended to provide context for understanding the patch series and to facilitate further discussion.
Welcome review and discussion.
Thanks,
Mingwei Jia