Re: UNDO with FILEOPS, and the FLUX table AM

Hello again.

v10 is attached. It is the same three commits as v9, with the fixes for
the four problems in my last message.

0001 cluster-wide UNDO engine (records in the WAL stream)
0002 transactional filesystem operations (FILEOPS) backed by UNDO
0003 route filesystem mutations through FILEOPS

- A loser whose UNDO all precedes the checkpoint is now rolled back
(Shihao's point 1; 0001, test_fileops/005). Each online checkpoint
writes an XLOG_UNDO_INFLIGHT record with the UNDO chain heads of the
transactions in flight, and redo starts tracking them from it.
InsertXactUndoData() holds off the checkpoint's redo-pointer phase
while it inserts a batch and publishes the head, so a batch cannot
fall between the two.

- A crash or error part way through CREATE DATABASE leaves no directory
behind, for either strategy (point 2; 0002 and 0003, recovery/066).
The new database directory, and the destination of copydir(), are
created with FileOpsMkdirTree(), whose UNDO removes the whole tree.

- An ordinary filesystem error is an ERROR again, not a PANIC (0002,
test_fileops/004). An immediate operation now makes its system call
first and WAL-logs it only once the call has succeeded. That leaves a
short window, between the call and its WAL record, in which a crash
leaves the change in place with nothing to reverse it, as without
FILEOPS; fileops.c and the README say so. The other order closes the
window only by making every filesystem error a PANIC.

- The abort record carries the transaction's FILEOPS inverses, and abort
redo applies them, so a standby, and the primary after a crash, end up
with the files the primary had after ROLLBACK (0002, test_fileops/008
and /010). This is the gap I described with v9. ROLLBACK PREPARED
carries them too, from the 2PC state. It applies them before it clears
DELAY_CHKPT_START, which RecordTransactionAbortPrepared() sets since
cea96cdfecb, so a checkpoint cannot move past the abort record while
they are not yet durable.

All of these were in v8, in the commit that v9 left out, and come from
there. Bringing them back also brought a few smaller FILEOPS changes v8
had made in the same place: FileOpsRename() renames at once and is
undone by renaming back (test_fileops/018); an inverse reverts only an
object that still shows the operation's effect, so a name reused since
is left alone; a rollback makes its inverses durable once per path
rather than once per inverse; and the abort data a transaction can
accumulate is capped, so the abort record itself cannot fail.

As I said, the subtransaction and two-phase paths stay. They support
future behavior, so strictly not necessary right now. Perhaps I should
separate them into a 4th patch to make that explicit.

Size

0001-0003 now add about 17,800 lines; counted as Shihao did (C outside
the tests, no comments or blank lines) that is about 6,800, of which
3,400 are in 0001. v9 was about 5,800. Most of the difference is in
fileops.c and fileops_undo.c but what this shows me is that I should
review 0001 to see if there are ways to break it down into morsel sized
bits easier to chew on.

Also, I wonder if there is a smaller table AM example I could come up
with that uses UNDO to bring that back in, or something else adjacent to
that so as to bring a few more layers back into the cake. Bad idea?
Too early? I feel that without that the series is less interesting but
with it too large. I'll think on this a bit.

Thanks for the continued interest, this is not a small feature to
consider.

best.

-greg