Re: BUG #19616: pgoutput sends stream abort ('A') to clients that did not enable streaming

Hi, Tyler!

Thanks for the report and analysis.

ReorderBufferTruncateTXN is used both after streaming and when discarding
already-aborted transactions at eviction. Streaming callers mark the
top-level xact before truncating. The abort-discard path does not mark
it. The attached patch marks a subxact as streamed only when the
top-level xact already is. A later abort then does not emit stream_abort
for XIDs that were never sent downstream.

Patch with regress tests attached.

чт, 13 авг. 2026 г. в 20:30, PG Bug reporting form <noreply(at)postgresql(dot)org>:

> The following bug has been logged on the website:
>
> Bug reference: 19616
> Logged by: Tyler Smart
> Email address: tyler(at)smarts(dot)io
> PostgreSQL version: 18.4
> Operating system: Linux (Docker; also Google Cloud SQL)
> Description:
>
> Since PostgreSQL 18, pgoutput can send a Stream Abort ('A') message to a
> client that connected with proto_version 1 and never enabled streaming.
> Protocol 1 clients do not implement the stream message set, so consumers
> fail on it: Debezium (and tools that embed it, like Airbyte) dies with
> "Unsupported message type: A", and because the crash repeats at the same
> WAL
> position on every restart, the slot stops advancing until
> max_slot_wal_keep_size invalidates it.
>
> I reproduced this on 18.4 (Debian 18.4-1.pgdg13+1, official Docker image)
> and on Cloud SQL 18.1. The identical script does not reproduce it on 17.10.
>
> The trigger needs three conditions in one decode run:
>
> 1. logical_decoding_work_mem is exceeded, so eviction runs.
> 2. The eviction candidate has already aborted in clog.
> 3. That transaction has a subtransaction with changes still in memory.
>
> From reading REL_18_STABLE, the cause appears to be commit 072ee847ad4
> ("Skip logical decoding of already-aborted transactions"). It added
> ReorderBufferCheckAndTruncateAbortedTXN, which discards aborted
> transactions
> at eviction time via ReorderBufferTruncateTXN. That function marks every
> subtransaction that still has in-memory changes as streamed
> (ReorderBufferMaybeMarkTXNStreamed, in the subtxn loop near
> reorderbuffer.c:1675) without checking whether the connection streams at
> all. The top-level transaction is handled correctly, since its marking
> happens at call sites guarded by the streaming flag. Only the
> subtransaction
> marking is unconditional.
>
> When decoding later reaches the abort record, ReorderBufferAbort sees
> rbtxn_is_streamed on the subtransaction and invokes the stream_abort
> callback (near reorderbuffer.c:3092). pgoutput_stream_abort is guarded only
> by assertions, so production builds write 'A' onto a proto_version 1
> stream. I expect a cassert build to fail Assert(rbtxn_is_streamed(toptxn))
> there instead, since the top-level transaction is not marked, though I have
> not verified that.
>
> client that connected with proto_version 1 and never enabled streaming.
> Protocol 1 clients do not implement the stream message set, so consumers
> fail on it: Debezium (and tools that embed it, like Airbyte) dies with
> "Unsupported message type: A", and because the crash repeats at the same
> WAL
> position on every restart, the slot stops advancing until
> max_slot_wal_keep_size invalidates it.
>
> I reproduced this on 18.4 (Debian 18.4-1.pgdg13+1, official Docker image)
> and on Cloud SQL 18.1. The identical script does not reproduce it on 17.10.
>
> The trigger needs three conditions in one decode run:
>
> 1. logical_decoding_work_mem is exceeded, so eviction runs.
> 2. The eviction candidate has already aborted in clog.
> 3. That transaction has a subtransaction with changes still in memory.
>
> From reading REL_18_STABLE, the cause appears to be commit 072ee847ad4
> ("Skip logical decoding of already-aborted transactions"). It added
> ReorderBufferCheckAndTruncateAbortedTXN, which discards aborted
> transactions
> at eviction time via ReorderBufferTruncateTXN. That function marks every
> subtransaction that still has in-memory changes as streamed
> (ReorderBufferMaybeMarkTXNStreamed, in the subtxn loop near
> reorderbuffer.c:1675) without checking whether the connection streams at
> all. The top-level transaction is handled correctly, since its marking
> happens at call sites guarded by the streaming flag. Only the
> subtransaction
> marking is unconditional.
>
> When decoding later reaches the abort record, ReorderBufferAbort sees
> rbtxn_is_streamed on the subtransaction and invokes the stream_abort
> callback (near reorderbuffer.c:3092). pgoutput_stream_abort is guarded only
> by assertions, so production builds write 'A' onto a proto_version 1
> stream. I expect a cassert build to fail Assert(rbtxn_is_streamed(toptxn))
> there instead, since the top-level transaction is not marked, though I have
> not verified that.
>
> Self-contained reproduction (the SQL decoding interface acts as a
> non-streaming client, so no replication client is needed):
>
> docker run -d -e POSTGRES_PASSWORD=pw postgres:18 -c wal_level=logical
>
> CREATE TABLE t(id int, filler text);
> CREATE PUBLICATION pub FOR TABLE t;
> SELECT pg_create_logical_replication_slot('s', 'pgoutput');
>
> BEGIN;
> SAVEPOINT sp;
> INSERT INTO t VALUES (0, 'subtransaction-change');
> RELEASE SAVEPOINT sp;
> INSERT INTO t SELECT g, repeat('x', 1000) FROM generate_series(1, 5000)
> g;
> ROLLBACK;
>
> INSERT INTO t VALUES (1, 'after');
>
> SET logical_decoding_work_mem = '64kB';
> SELECT chr(get_byte(data,0)) AS msgtype, count(*)
> FROM pg_logical_slot_peek_binary_changes('s', NULL, NULL,
> 'proto_version','1','publication_names','pub')
> GROUP BY 1 ORDER BY 2 DESC;
>
> Actual output on 18.4:
>
> msgtype | count
> ---------+-------
> B | 1
> R | 1
> C | 1
> I | 1
> A | 1
>
> Expected: no A row. A proto_version 1 client must never receive stream
> messages, and an aborted transaction should produce no output at all.
> PG 17.10 produces the expected output with the same script, as does 18.4
> when logical_decoding_work_mem is raised enough that eviction never fires.
>
> Impact: any protocol 1 consumer on a busy PG 18 server can hit this with a
> single canceled or deadlocked transaction that used savepoints, decoded
> while the buffer is past logical_decoding_work_mem. We hit it in production
> through Debezium, where the retry loop pinned the slot until Postgres
> invalidated it. Raising logical_decoding_work_mem only lowers the
> probability.
>
> Suggested direction: ReorderBufferTruncateTXN should mark subtransactions
> as streamed only when truncating on behalf of streaming, the same way the
> top-level marking is already gated, or the abort-discard path should skip
> the marking entirely.
>
>
>
>
>