Re: FATAL: could not send end-of-streaming message to primary: no COPY in progress

From: Stepan Neretin <slpmcf(at)gmail(dot)com>
To: Zsolt Parragi <zsolt(dot)parragi(at)percona(dot)com>
Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org, Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
Subject: Re: FATAL: could not send end-of-streaming message to primary: no COPY in progress
Date: 2026-11-18 04:19:17
Message-ID: CA+Yyo5QfF35o42eBN=CeeN9HpUSfugCjyHw6rW-x8ZJgrnXNZw@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Wed, Nov 18, 2026 at 4:54 AM Zsolt Parragi <zsolt(dot)parragi(at)percona(dot)com>
wrote:

> Hello
>
> 1.
>
> > Subject: [PATCH v2 2/2] Add TAP test for primary shutdown during
> streaming
>
> This should also have a meson.build edit so the meson builds also run
> the new test
>
> 2. Stopping the primary and waiting a bit gives this on the standby:
>
> LOG: replication terminated by primary server
> DETAIL: End of WAL reached on timeline 1 at 0/030000B0.
> FATAL: could not receive database system identifier and timeline ID
> from the primary server: no connection to the server
>
> After the early return walreceiver waits for new orders and then runs
> IDENTIFY_SYSTEM on the closed connection. The test passes because it
> only checks that the old message is gone.
>
> 3. On master, a clean publisher shutdown hits the same error in the
> logical apply worker, and that disables a disable_on_error
> subscription:
>
> LOG: data stream from publisher has ended
> ERROR: could not send end-of-streaming message to primary: server
> closed the connection unexpectedly
> LOG: subscription "sub" has been disabled because of an error
>
> With the patch only the first line remains and the subscription stays
> enabled. This doesn't seem like a bad thing, but probably should be
> mentioned at least in the commit message, and maybe tested too?
>
> 4. Partial WAL segment archived on archive_mode=always standbys
>
> Since endstreaming no longer fails, walreceiver continues into the
> end-of-timeline block in WalReceiverMain(), which closes the current
> segment and calls XLogArchiveNotify() on it, even though the segment
> is incomplete.
>
> Setup:
>
> * standby with archive_mode = always and archive_command = 'test ! -f
> arch/%f && cp %p arch/%f'
> * stop the primary
> * wait a few seconds
> * start it again
> * run pg_switch_wal() a few times:
>
> master: archived while primary down: (none)
> pg_stat_archiver: 5|000000010000000000000007|0|
> patched: archived while primary down: 000000010000000000000003
> pg_stat_archiver:
> 1|000000010000000000000003|6|000000010000000000000003
>
> 5. Would something like this work instead?
>
> libpqrcv_endstreaming():
> + *next_tli = 0;
> + if (PQstatus(conn->streamConn) == CONNECTION_BAD)
> + return;
> walreceiver.c, after walrcv_endstreaming(wrconn, &primaryTLI):
> + if (primaryTLI == 0)
> + proc_exit(0);
>

Hi Zsolt,

Thanks for the careful review. Agreed on points 1 to 4. I missed those
follow ons in v2. On point 5 I like the direction, but CONNECTION_BAD
alone does not cover the original report.

After COPY 0, libpq has left COPY while PQstatus() can still be
CONNECTION_OK, so PQputCopyEnd() still fails with no COPY in progress.
That is why copyEnded stays. CONNECTION_BAD is a useful complement for
an abrupt close, so I kept that too.

v3 also does proc_exit(0) in physical walreceiver when endstreaming
reports next_tli == 0. Timeline switch still sends a non zero next TLI
and keeps waiting mode. Clean shutdown does not, and falling through
was what caused the IDENTIFY_SYSTEM FATAL and the incomplete segment
archive under archive_mode=always.

Both TAP tests are in meson.build now, and they cover the endstreaming
FATAL, IDENTIFY_SYSTEM, disable_on_error, incomplete segment archive,
and reconnect after restart.

Attached.

Regards,
Stepan Neretin

Attachment Content-Type Size
v3-0002-Add-TAP-tests-for-primary-publisher-shutdown-during-.patch text/x-patch 8.2 KB
v3-0001-Fix-FATAL-when-primary-ends-streaming-with-COPY-0.patch text/x-patch 4.2 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Fujii Masao 2026-11-18 04:34:24 Re: pg_basebackup: check timeline history file write failures in ReceiveXlogStream()
Previous Message Ajin Cherian 2026-11-18 04:17:59 Re: Initial table sync copies the whole table when it is no longer published