BUG #19616: pgoutput sends stream abort ('A') to clients that did not enable streaming

The following bug has been logged on the website:

Bug reference: 19616
Logged by: Tyler Smart
Email address: tyler(at)smarts(dot)io
PostgreSQL version: 18.4
Operating system: Linux (Docker; also Google Cloud SQL)
Description:

Since PostgreSQL 18, pgoutput can send a Stream Abort ('A') message to a
client that connected with proto_version 1 and never enabled streaming.
Protocol 1 clients do not implement the stream message set, so consumers
fail on it: Debezium (and tools that embed it, like Airbyte) dies with
"Unsupported message type: A", and because the crash repeats at the same WAL
position on every restart, the slot stops advancing until
max_slot_wal_keep_size invalidates it.

I reproduced this on 18.4 (Debian 18.4-1.pgdg13+1, official Docker image)
and on Cloud SQL 18.1. The identical script does not reproduce it on 17.10.

The trigger needs three conditions in one decode run:

1. logical_decoding_work_mem is exceeded, so eviction runs.
2. The eviction candidate has already aborted in clog.
3. That transaction has a subtransaction with changes still in memory.

From reading REL_18_STABLE, the cause appears to be commit 072ee847ad4
("Skip logical decoding of already-aborted transactions"). It added
ReorderBufferCheckAndTruncateAbortedTXN, which discards aborted transactions
at eviction time via ReorderBufferTruncateTXN. That function marks every
subtransaction that still has in-memory changes as streamed
(ReorderBufferMaybeMarkTXNStreamed, in the subtxn loop near
reorderbuffer.c:1675) without checking whether the connection streams at
all. The top-level transaction is handled correctly, since its marking
happens at call sites guarded by the streaming flag. Only the subtransaction
marking is unconditional.

When decoding later reaches the abort record, ReorderBufferAbort sees
rbtxn_is_streamed on the subtransaction and invokes the stream_abort
callback (near reorderbuffer.c:3092). pgoutput_stream_abort is guarded only
by assertions, so production builds write 'A' onto a proto_version 1
stream. I expect a cassert build to fail Assert(rbtxn_is_streamed(toptxn))
there instead, since the top-level transaction is not marked, though I have
not verified that.

client that connected with proto_version 1 and never enabled streaming.
Protocol 1 clients do not implement the stream message set, so consumers
fail on it: Debezium (and tools that embed it, like Airbyte) dies with
"Unsupported message type: A", and because the crash repeats at the same WAL
position on every restart, the slot stops advancing until
max_slot_wal_keep_size invalidates it.

I reproduced this on 18.4 (Debian 18.4-1.pgdg13+1, official Docker image)
and on Cloud SQL 18.1. The identical script does not reproduce it on 17.10.

The trigger needs three conditions in one decode run:

1. logical_decoding_work_mem is exceeded, so eviction runs.
2. The eviction candidate has already aborted in clog.
3. That transaction has a subtransaction with changes still in memory.

From reading REL_18_STABLE, the cause appears to be commit 072ee847ad4
("Skip logical decoding of already-aborted transactions"). It added
ReorderBufferCheckAndTruncateAbortedTXN, which discards aborted transactions
at eviction time via ReorderBufferTruncateTXN. That function marks every
subtransaction that still has in-memory changes as streamed
(ReorderBufferMaybeMarkTXNStreamed, in the subtxn loop near
reorderbuffer.c:1675) without checking whether the connection streams at
all. The top-level transaction is handled correctly, since its marking
happens at call sites guarded by the streaming flag. Only the subtransaction
marking is unconditional.

When decoding later reaches the abort record, ReorderBufferAbort sees
rbtxn_is_streamed on the subtransaction and invokes the stream_abort
callback (near reorderbuffer.c:3092). pgoutput_stream_abort is guarded only
by assertions, so production builds write 'A' onto a proto_version 1
stream. I expect a cassert build to fail Assert(rbtxn_is_streamed(toptxn))
there instead, since the top-level transaction is not marked, though I have
not verified that.

Self-contained reproduction (the SQL decoding interface acts as a
non-streaming client, so no replication client is needed):

docker run -d -e POSTGRES_PASSWORD=pw postgres:18 -c wal_level=logical

CREATE TABLE t(id int, filler text);
CREATE PUBLICATION pub FOR TABLE t;
SELECT pg_create_logical_replication_slot('s', 'pgoutput');

BEGIN;
SAVEPOINT sp;
INSERT INTO t VALUES (0, 'subtransaction-change');
RELEASE SAVEPOINT sp;
INSERT INTO t SELECT g, repeat('x', 1000) FROM generate_series(1, 5000) g;
ROLLBACK;

INSERT INTO t VALUES (1, 'after');

SET logical_decoding_work_mem = '64kB';
SELECT chr(get_byte(data,0)) AS msgtype, count(*)
FROM pg_logical_slot_peek_binary_changes('s', NULL, NULL,
'proto_version','1','publication_names','pub')
GROUP BY 1 ORDER BY 2 DESC;

Actual output on 18.4:

msgtype | count
---------+-------
B | 1
R | 1
C | 1
I | 1
A | 1

Expected: no A row. A proto_version 1 client must never receive stream
messages, and an aborted transaction should produce no output at all.
PG 17.10 produces the expected output with the same script, as does 18.4
when logical_decoding_work_mem is raised enough that eviction never fires.

Impact: any protocol 1 consumer on a busy PG 18 server can hit this with a
single canceled or deadlocked transaction that used savepoints, decoded
while the buffer is past logical_decoding_work_mem. We hit it in production
through Debezium, where the retry loop pinned the slot until Postgres
invalidated it. Raising logical_decoding_work_mem only lowers the
probability.

Suggested direction: ReorderBufferTruncateTXN should mark subtransactions
as streamed only when truncating on behalf of streaming, the same way the
top-level marking is already gated, or the abort-discard path should skip
the marking entirely.