Re: UNDO with FILEOPS, and the FLUX table AM

Hi Greg,

On Sun, Oct 25, 2026 at 7:12 PM Greg Burd <greg(at)burd(dot)me> wrote:
> - The FILEOPS WRITE inverse treated a failed stat() of its file like a
> missing file and reported success, so a crash loser's overwritten
> bytes were never restored.

I tested this with a real EACCES instead of the injection point. A
crash loser overwrites the first 10 bytes of a file, and before the
restart the file's directory loses its search permission (the server
does not run as root).
v7: no ATM entry, the bytes stay overwritten (3/3 runs).
v8: the loser goes to the ATM, and once the directory is readable
again the worker restores the bytes (3/3).

That only holds while the error is short. If the directory is still
unreadable when the revert worker starts (I held it for 10 s), the
worker walks the chain 3 times, about 1 s and 2 s apart, logs
"giving up on 1 UNDO record(s) of transaction N that failed to apply
3 times" and forgets the entry. The bytes stay overwritten, as in v7
(3/3). So at the default logical_revert_naptime, an EACCES or EIO
that lasts more than about 3 s ends the way it did before, now with
a WARNING. Is that intended for a record that failed to apply? From
reading process_revert_entry(), LOGICAL_REVERT_MAX_RECORD_RETRIES
gives up on it the same way as on a malformed chain.

> - The logical revert launcher gave up on a database whose worker could
> not be registered (no free background-worker slot): it recorded the
> database as served anyway, so its ATM entries were never reverted and
> their WAL was kept for good.

I tested this with real slot exhaustion. worker_spi workers take
every free slot, an ATM entry is queued the way test_fileops/019
does it, and the launcher is woken and fails to register. Then the
worker_spi workers are terminated and the launcher is woken again.
v7: the entry is not drained after 90 s (past one 60 s scan), no
revert worker runs, and its WAL segment is still kept after 4
segment switches and a CHECKPOINT (3/3).
v8: the launcher retries and the entry is drained (3/3).

> - When the revert worker's 64-slot retry table was full, a chain that
> failed to apply was given up on its first failure, marked reverted
> and forgotten.

Your test_idxundo/020 leaves 64 of 65 entries after one pass on v7
and fails (3/3). On v8 it leaves 65 of 65 and passes (3/3).

Each check also fails on v8 with only its fix reverted (3/3 each):
my two tests, your test_fileops/020 and your test_idxundo/020.

One thing for the 0001/0002/0005 post: the tests for the launcher fix
and the retry-table fix (test_idxundo/021 and 020) are in 0006, and
they need idxundo_atm_add() or an index_undo table. Without 0006
those two fixes have no test. The attached nocfbot diff adds my two
tests to test_fileops as t/021 and t/022. They only need
test_fileops, injection_points and worker_spi, so they apply on top
of 0005. I have not ported test_idxundo/020. t/021 takes HOLD=<s> in
the environment (default 0) for the 10 s case, which fails on v8.

On master dfd7ede529f all 21 patches apply. 0001-0005 together
build with no warnings (cassert), and these pass: make check (239),
test_fileops (19 files, including 019, 020 and the two new ones),
and recovery 061-064, 071, 072 and 090.

Limits: cassert only, Linux, one machine, no timings. v7 here is the
full v7 series. fileops_undo.c and logical_revert_worker.c are the
same in v7-0005, except for one whitespace line.

Regards,
Manu