# UNDO/FILEOPS v3/v4 review: evidence (2026-10-13)

Target: **v4** (Greg, 2026-10-13 00:27, "Re: UNDO with FILEOPS, and the FLUX
table AM"), which supersedes v3 (2026-10-12) one day later; 0002, 0003 and
0006-0008 are byte-identical between v3 and v4 (see PATCH_MAP.md). Applied with
`git am` on master **b28a2b2bfa5** (2026-10-13), all 20 patches clean. Trees in
~/.cache/ctr-v3: normal (-O2), cassert (-O2 + asserts, same flags as the v2
install ~/undo-v2-inst used for the v2 numbers), asan (ASan+UBSan, gcc 16).
Machine: Ryzen AI 9 HX 370, NVMe + btrfs. Lane A CPUs, integrity sentinel clean.

## Finding 1 (measured, new in v3): ROLLBACK and crash recovery pay 2 fsyncs per FILEOPS UNDO record

v3 added `fileops_sync_inverse()` (fileops_undo.c ~697): after **every**
applied inverse it fsyncs the file **and** its parent directory (elevel PANIC).
The same apply path runs inline at ROLLBACK (FILEOPS can no longer be deferred,
`needs_inline_abort`, xactundo.c) and in the crash-recovery undo phase before
the cluster opens. N inverses on the same file -> 2N fsyncs of the same two
inodes.

A) Client-visible ROLLBACK latency, N chmods in one txn, only ROLLBACK timed,
   median of 3, every trial validated (file mode back to baseline). Same
   workload as our v2 report (logs/bench/rollback_raw.tsv, summary.txt):

   threshold    build   N=200    N=1000    N=8000     N=32000   (ms)
   65536        v2      0.64     2.47      19.6       77.0
   65536        v4c     132      660       5822       22851
   1e9 (inline) v4c     147      752       5423       21748
   1 (defer)    v4c     154      738       5870       23007

   ~0.7 ms per record (= this NVMe's fsync cost) vs ~2.4 us in v2: ~300x.
   undo_instant_abort_threshold no longer changes anything for FILEOPS (by
   design in v3: FILEOPS UNDO is always inline).

B) Discriminator (logs/fsync_discriminator.txt), N=1000, median of 3:

   build   fsync=on   fsync=off
   v2      1.9 ms     1.8 ms
   v4n     638 ms     1.4 ms      <- no asserts: the whole cost is fsync
   v4c     1006 ms    1.5 ms

C) Crash with an in-flight txn of N FILEOPS records, immediate stop, restart;
   undo phase timed from the server's own log; undo phase completes before
   "ready to accept connections" in every trial; records applied == N
   (logs/bench/recovery_raw.tsv, recovery_v4_raw.tsv):

   N       v2        v4n        v4c
   2000    2 ms      1318 ms    1450 ms
   8000    7-8 ms    5570 ms    5640 ms
   32000   28-39 ms  (~22 s extrapolated, not run)

   So the v2 point "linear before open" still holds, with a ~700x larger slope.
   Forward cost (doing the N chmods) is ~0.9 ms/op in BOTH v2 and v4
   (logs/forward_cost.txt): the regression is entirely on the undo side.
   Redo of the same records is also fsync-bound: a crash during ROLLBACK of
   6000 records spent 7.7 s in redo (ASan build, logs/adv/S5b.server.log).

Why it matters / what to ask: durability of the inverse before the
abort/CLR is the point of the v3 fix, but it can be batched: fsync each
distinct path (and parent) once per chain, before the record that marks the
chain done, instead of once per record. As is, a txn that touched one file
10k times takes ~7 s to ROLLBACK and adds ~7 s of undo before open after a
crash. Repro: scripts/repro_rollback_fsync.sql (+ fsync_discriminator.sh).

## Finding 2 (measured): "every prefix builds and passes its own tests" does not hold

Clean checkout + clean configure (cassert, TAP, injection points) per prefix,
core `make check`, then the TAP/regress files the patch adds or modifies
(scripts/prefix_check.sh, logs/prefix/). Every prefix 0001-0008 **builds with 0
warnings and passes core make check (239/239)**. Own tests:

- 0001: PASS (pg_waldump/001).
- 0002: **FAIL** test_fileops/t/001_fileops_apply_failure.pl: sets
  `max_logical_revert_workers = 0`, a GUC added in 0005 -> "unrecognized
  configuration parameter", pg_ctl start fails. Same shape as Shihao's #1
  (059/061 setting a 0005 GUC). Not seen by the build farm/meson because
  test_fileops is not listed in src/test/modules/{Makefile,meson.build} until
  0005 (logs/meson_audit.txt). Recovery 060-063, 072: PASS.
- 0003: no own tests; build + core PASS.
- 0004: **FAIL** 6 tests die with `access method "flux" does not exist`
  (FLUX = 0019): recovery 073, 074, 075, 083, 089 and test_checksums/025 (all
  registered in meson at this prefix). recovery/114 needs
  shared_preload_libraries=test_idxundo (module added in 0006): pg_ctl start
  fails and prove bails out (115 [also FLUX], 117, 152 not reached).
  085/096 fail on pg_walinspect/amcheck not installed under the autoconf
  harness (recovery Makefile EXTRA_INSTALL not extended; meson installs all
  contrib) - minor, autoconf-only.
- 0005: **8 orphan TAP files**: test_idxundo/t/002-009 ship here but the module
  (Makefile, meson.build, .c) arrives in 0006 - nothing can run them.
  test_fileops/005 dies: reloption `index_undo` (0006). **test_fileops/016
  FAILS 3/5 and shows real misbehaviour at this prefix**: with an injected
  fsync failure on an inverse during abort, the backend raises ERROR inside
  abort, re-enters abort 5 times (5 abort records) and ends in
  `PANIC: ERRORDATA_STACK_SIZE exceeded` instead of one clean PANIC. The
  elevel=PANIC for that path is only introduced in 0007 (the hash patch).
  On the full series 005 and 016 PASS (verified).
- 0006, 0007: build + core PASS (own tests not run in this pass).
- 0008: **FAIL** recovery 094 and 121 (registered in meson here) die with
  `access method "flux" does not exist`.

Repro: scripts/repro_prefix_tests.sh (0002 and 0004 cases).

## Adversarial pass (ASan+UBSan, crash injection): nothing found

scripts/adv_fileops.sh on inst-asan. Victim txn = 13 mixed FILEOPS ops
(chmod, write, truncate, create, mkdir, create-in-new-dir, link, symlink,
setxattr, removexattr, rename) + NPAD chmods to widen windows. Invariant: test
tree (type, mode, size, md5, link target, user xattrs) == committed baseline;
server log has no PANIC/TRAP/assert; no ASan/UBSan report.

  S0 negative control (same ops COMMITTED)            DIVERGED (harness valid)
  S1 plain ROLLBACK                                    OK
  S2 SAVEPOINT / ROLLBACK TO + more ops, ROLLBACK      OK
  S3 in-flight, postmaster immediate stop              OK (3011 records undone before open)
  S4 in-flight, kill -9 backend (crash-restart)        OK (3011 records)
  S5 kill -9 backend DURING its ROLLBACK               OK ("Failed process was running: ROLLBACK")
  S5b same, snapshot at FIRST accepted connection      already reverted at open (redo finished it)
  S6 crash DURING the recovery undo phase, restart     OK (1.8 s into phase; 2nd recovery re-applies
                                                       the whole chain, 6011 records, idempotent)
  S7 PREPARE TRANSACTION, crash, ROLLBACK PREPARED     OK (prepared xact skipped by undo, kept)

No sanitizer report in any scenario. (The 6 -Wnonnull warnings of the ASan
build are gcc-16 false positives in upstream code: path.c, indexcmds.c,
tablecmds.c, md.c.) Also confirmed by construction: v3's alignment bug (undo
record headers at non-MAXALIGN'd offsets) is what v4 fixes; we tested v4.

Harness notes (cost us time, not findings): FILEOPS rename is deferred to
commit, so writing to the new name inside the txn errors out (the first S1/S2
"OK" were vacuous until S0 caught it); `pg_ctl start` waits by default (-W
needed to crash inside the undo phase).

## Not done in this pass
- 0006/0007 own TAP suites; 0005 recovery 090/126/134 and test_fileops
  002-009 at that prefix.
- Deeper adversarial (ENOSPC/EIO during inline apply -> ATM fallback path at
  xactundo.c ~1183, CREATE DATABASE crash windows, standby).
- v2-vs-v4 ABBA second round (effect size 300x makes it moot).