[PATCH]Fix pg_xact corruption from subtransaction abort after subcommit

Greetings,

The subtransaction commit path can corrupt pg_xact, and the offending code
is all stock.  A subtransaction that has already subcommitted can still be
forced to abort, and when that happens its XID is left in the parent's list
of committed children while pg_xact records it as aborted.  When the parent
commits, TransactionIdSetTreeStatus() tries to mark that aborted XID
committed.  With assertions on, that's a TRAP in clog.c; with them off, it
writes the wrong status, and since the parent's commit record carries the
same child list, replay hits it too and recovery never finishes.

The window is in CommitSubTransaction():

    s->state = TRANS_COMMIT;
    ...
    if (FullTransactionIdIsValid(s->fullTransactionId))
        AtSubCommit_childXids();
    AfterTriggerEndSubXact(true);
    AtSubCommit_Portals(...);
    ...

By the time AfterTriggerEndSubXact() runs, AtSubCommit_childXids() has
already copied our XID and any committed grandchildren into the parent's
array.  If one of the later steps throws (OOM being the obvious case),
control longjmps into AbortSubTransaction() with the subtransaction still in
TRANS_COMMIT state.  We record the XID aborted and leave it sitting in the
parent's list; AtSubAbort_childXids() only frees our own array, not the
parent's.

An error thrown at a subtransaction's commit inside a PL/pgSQL EXCEPTION 
block is caught right there, so the subtransaction aborts while the 
surrounding transaction goes on to commit.  That is the shape that bites.  
A subtransaction-commit callback that raises on 
SUBXACT_EVENT_COMMIT_SUB reproduces it with no core changes and no 
injection points; an injection point just after AtSubCommit_childXids() 
does too:

    BEGIN;
    DO $$
    BEGIN
        BEGIN
            INSERT INTO t VALUES (1);   -- subtransaction acquires an XID
        EXCEPTION WHEN OTHERS THEN
            NULL;                       -- swallow the commit-time error
        END;
    END $$;
    COMMIT;                             -- crashes here

    TRAP: failed Assert("curval == 0 || ... || curval == status"),
          File: "clog.c", Line: 702
        TransactionIdSetStatusBit
        TransactionIdSetTreeStatus
        TransactionIdCommitTree
        RecordTransactionCommit

The fix is small and stays in xact.c.  AtSubCommit_childXids() records the
parent's child count before it appends, and AbortSubTransaction() restores
that count when the subtransaction aborts after the transfer.  The entries
we added are the tail of the parent's array, so restoring the length drops
exactly them; the grandchildren revert to aborted, which is correct because
the whole subtree is rolling back.

I confirmed on current master that the reproduction crashes without the
patch and commits cleanly with it, that the aborted row is gone, that a
committed sibling savepoint survives, and that the regression tests pass.

--
Bryan Green
EDB: https://www.enterprisedb.com