| From: | Stepan Neretin <slpmcf(at)gmail(dot)com> |
|---|---|
| To: | Zsolt Parragi <zsolt(dot)parragi(at)percona(dot)com> |
| Cc: | pgsql-hackers(at)lists(dot)postgresql(dot)org, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> |
| Subject: | Re: FATAL: could not send end-of-streaming message to primary: no COPY in progress |
| Date: | 2026-11-18 04:19:17 |
| Message-ID: | CA+Yyo5QfF35o42eBN=CeeN9HpUSfugCjyHw6rW-x8ZJgrnXNZw@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
On Wed, Nov 18, 2026 at 4:54 AM Zsolt Parragi <zsolt(dot)parragi(at)percona(dot)com>
wrote:
> Hello
>
> 1.
>
> > Subject: [PATCH v2 2/2] Add TAP test for primary shutdown during
> streaming
>
> This should also have a meson.build edit so the meson builds also run
> the new test
>
> 2. Stopping the primary and waiting a bit gives this on the standby:
>
> LOG: replication terminated by primary server
> DETAIL: End of WAL reached on timeline 1 at 0/030000B0.
> FATAL: could not receive database system identifier and timeline ID
> from the primary server: no connection to the server
>
> After the early return walreceiver waits for new orders and then runs
> IDENTIFY_SYSTEM on the closed connection. The test passes because it
> only checks that the old message is gone.
>
> 3. On master, a clean publisher shutdown hits the same error in the
> logical apply worker, and that disables a disable_on_error
> subscription:
>
> LOG: data stream from publisher has ended
> ERROR: could not send end-of-streaming message to primary: server
> closed the connection unexpectedly
> LOG: subscription "sub" has been disabled because of an error
>
> With the patch only the first line remains and the subscription stays
> enabled. This doesn't seem like a bad thing, but probably should be
> mentioned at least in the commit message, and maybe tested too?
>
> 4. Partial WAL segment archived on archive_mode=always standbys
>
> Since endstreaming no longer fails, walreceiver continues into the
> end-of-timeline block in WalReceiverMain(), which closes the current
> segment and calls XLogArchiveNotify() on it, even though the segment
> is incomplete.
>
> Setup:
>
> * standby with archive_mode = always and archive_command = 'test ! -f
> arch/%f && cp %p arch/%f'
> * stop the primary
> * wait a few seconds
> * start it again
> * run pg_switch_wal() a few times:
>
> master: archived while primary down: (none)
> pg_stat_archiver: 5|000000010000000000000007|0|
> patched: archived while primary down: 000000010000000000000003
> pg_stat_archiver:
> 1|000000010000000000000003|6|000000010000000000000003
>
> 5. Would something like this work instead?
>
> libpqrcv_endstreaming():
> + *next_tli = 0;
> + if (PQstatus(conn->streamConn) == CONNECTION_BAD)
> + return;
> walreceiver.c, after walrcv_endstreaming(wrconn, &primaryTLI):
> + if (primaryTLI == 0)
> + proc_exit(0);
>
Hi Zsolt,
Thanks for the careful review. Agreed on points 1 to 4. I missed those
follow ons in v2. On point 5 I like the direction, but CONNECTION_BAD
alone does not cover the original report.
After COPY 0, libpq has left COPY while PQstatus() can still be
CONNECTION_OK, so PQputCopyEnd() still fails with no COPY in progress.
That is why copyEnded stays. CONNECTION_BAD is a useful complement for
an abrupt close, so I kept that too.
v3 also does proc_exit(0) in physical walreceiver when endstreaming
reports next_tli == 0. Timeline switch still sends a non zero next TLI
and keeps waiting mode. Clean shutdown does not, and falling through
was what caused the IDENTIFY_SYSTEM FATAL and the incomplete segment
archive under archive_mode=always.
Both TAP tests are in meson.build now, and they cover the endstreaming
FATAL, IDENTIFY_SYSTEM, disable_on_error, incomplete segment archive,
and reconnect after restart.
Attached.
Regards,
Stepan Neretin
| Attachment | Content-Type | Size |
|---|---|---|
| v3-0002-Add-TAP-tests-for-primary-publisher-shutdown-during-.patch | text/x-patch | 8.2 KB |
| v3-0001-Fix-FATAL-when-primary-ends-streaming-with-COPY-0.patch | text/x-patch | 4.2 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Fujii Masao | 2026-11-18 04:34:24 | Re: pg_basebackup: check timeline history file write failures in ReceiveXlogStream() |
| Previous Message | Ajin Cherian | 2026-11-18 04:17:59 | Re: Initial table sync copies the whole table when it is no longer published |