| From: | Colin McDonnell <colinmcd94(at)gmail(dot)com> |
|---|---|
| To: | Zhijie Hou <houzhijie22(at)gmail(dot)com> |
| Cc: | Álvaro Herrera <alvherre(at)kurilemu(dot)de>, cca5507 <cca5507(at)qq(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Ajin Cherian <itsajin(at)gmail(dot)com>, Bertrand Drouvot <bertranddrouvot(dot)pg(at)gmail(dot)com>, pgsql-hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org> |
| Subject: | Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state |
| Date: | 2026-11-17 02:06:20 |
| Message-ID: | CALdYYtSX=Tuft9W7_rBNw2f8mRoQcXjui8YH4yLun4sLp0J_aw@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi,
I hit this from the other direction, making logical slots while other
sessions ran DDL, and only found this thread afterwards. So here is some
review by testing, of v7 plus Zhijie's two top-up patches on master at
72b9498df9.
The attached slot_under_ddl.sh needs only pgbench, psql and
test_decoding. Six pgbench clients create a table, add a column, insert
into it in a transaction held open for 2ms, and drop it. One session
loops over creating a slot, decoding twice and dropping it. 300 rounds
took about 20 seconds here.
It also counts decoded inserts that lack the added column, since nothing
else reports those. That is the silent case Manu described in September,
and it turns out about as common as the errors. 1000 rounds,
assert-enabled builds:
errors inserts missing the column
REL_14_STABLE f72e4686e2 156 129
REL_18_STABLE 813a74bc5a 188 143
REL_19_STABLE 0b1eb8b917 180 128
master 72b9498df9 168 150
The errors are "could not map filenumber" (filenode on 14) and
"pg_attribute catalog is missing N attribute(s)". One failed round can
count twice because it decodes twice. This was a Mac with load average
around 60 from unrelated work and slots being created in a tight loop,
so I wouldn't read anything into the rate.
master with v7 and the top-ups gave 0 errors and 0 missing columns in
5000 rounds.
The wrong output also isn't limited to transactions in flight at the
consistent point, or to missing columns. The attached
snapshot_build_stale.spec needs no load. ALTER TABLE t ADD COLUMN b and
TRUNCATE nk commit in BUILDING_SNAPSHOT, then inserts into t run across
and after CONSISTENT, one of them after a CHECKPOINT, followed by a
DELETE from nk, which has no replica identity, then an ANALYZE and one
more insert. On master with autovacuum off, every run:
table public.t: INSERT: id[integer]:1 a[text]:'straddles'
table public.t: INSERT: id[integer]:2 a[text]:'after consistent'
table public.t: INSERT: id[integer]:3 a[text]:'after checkpoint'
table public.t: INSERT: id[integer]:4 a[text]:'after analyze' b[integer]:5
With the patches all four inserts have b, and this line appears before
the last one:
table public.nk: DELETE: (no-tuple-data)
So the DELETE is lost on master with no error. From reading the code,
SnapBuildProcessRunningXacts moves builder->xmin but keeps the cached
builder->snapshot, and SnapBuildProcessChange hands that to new
transactions until a catalog-changing commit is decoded, which here is
the ANALYZE. With no later DDL or ANALYZE I don't see what would end it.
The DELETE goes because the truncated table's new file maps to no
relation under that snapshot, and a change with no tuple data takes the
goto change_done branch in ReorderBufferProcessTXN that exists for
mapped catalogs, instead of raising the error. I didn't try this through
pgoutput, but that skip comes before any plugin is called.
The attached ddl_mix.sql is harsher: DDL in a subtransaction after the
top-level xid is assigned, COMMIT PREPARED, ROLLBACK TO, a table rewrite
and a rolled back CREATE. It is the script's fourth argument. 3000
rounds:
errors inserts missing the column
master 677 250
master + v7 only 97 158
master + v7 + top-ups 0 0
v7 alone was clean on the simple workload, so this agrees with Zhijie's
review that it isn't enough. I didn't work out which of the three top-up
changes accounts for what. With two_phase => true on the slots (a small
edit to the script), master gave 879 and 338, and the patched build 0
and 0.
I had also written Sawada-san's April 2025 idea as a quick fix before
finding the thread: call SnapBuildCommitTxn from xact_decode in
BUILDING_SNAPSHOT and treat XACT_XINFO_HAS_INVALS as a catalog change
before CONSISTENT, with no NEW_CID tracking. Clean on the simple
workload, 80 errors and 144 missing columns on the mix, and it fails the
second and third permutations of Zhijie's spec. Mine had neither the
subxact check nor the accumulated invalidations from the top-ups, so
that may say more about my version than the idea, but with Manu's
numbers on large objects and user catalog tables I'd stop looking in
that direction.
Zhijie's snapshot_build spec was stable for me: 30 of 30 runs pass with
the top-ups, and on master all three permutations fail in 30 of 30. It
doesn't have Manu's ALTER TABLE ADD COLUMN and COMMIT PREPARED
permutations, which seem worth merging in.
With the patches, make check passes in contrib/test_decoding,
src/test/subscription and src/test/recovery. I did not run check-world.
On back branches, the patches don't apply to REL_18_STABLE with git
apply and one hunk needed doing by hand, because the NEW_CID case has a
different shape there. That diff is attached with the code changes only.
On it, 5000 rounds of the simple workload were clean, the spec passed 5
of 5 and test_decoding passes. I haven't run the subscription or
recovery suites on it or tried 14 through 17. I see no change to the
on-disk snapshot format or to a function signature, so conflicts look
like the main work of a backpatch, but someone who knows this code
should say.
Not covered by any of this:
- pgoutput and subscribers, I only looked at test_decoding output
- user catalog tables, which test_decoding never reads
- fast-forward mode, so top-up item 3 isn't exercised
- the cost, which I didn't measure. From reading it, a catalog-changing
transaction in BUILDING_SNAPSHOT now gets a ReorderBufferTXN, its
invalidations and a slot in committed.xip, as it already does in
FULL_SNAPSHOT.
Regards,
Colin
| Attachment | Content-Type | Size |
|---|---|---|
| slot_under_ddl.sh | text/plain | 2.7 KB |
| ddl_mix.sql | text/plain | 1.4 KB |
| snapshot_build_stale.spec | text/plain | 1.8 KB |
| nocfbot-REL_18_STABLE-v7-plus-topup.diff.txt | text/plain | 4.8 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Denis Smirnov | 2026-11-17 02:26:23 | Re: Fold NOT IN / <> ALL expressions containing NULL to FALSE |
| Previous Message | shihao zhong | 2026-11-17 02:04:46 | Re: CREATE TABLE LIKE with INHERITS skips checks on generated columns |