Re: Exit walsender before confirming remote flush in logical replication

> On Apr 7, 2026, at 13:39, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>
> On Tue, Apr 7, 2026 at 12:32 AM Andres Freund <andres(at)anarazel(dot)de> wrote:
>> Failed on CI just now:
>>
>> https://cirrus-ci.com/task/6745359004729344?logs=test_world#L410
>> https://api.cirrus-ci.com/v1/artifact/task/6745359004729344/testrun/build/testrun/subscription/038_walsnd_shutdown_timeout/log/regress_log_038_walsnd_shutdown_timeout
>>
>> [14:58:26.146](0.066s) ok 3 - have walreceiver pid 13796
>> ### Stopping node "publisher" using mode fast
>> # Running: pg_ctl --pgdata /home/postgres/postgres/build/testrun/subscription/038_walsnd_shutdown_timeout/data/t_038_walsnd_shutdown_timeout_publisher_data/pgdata --mode fast stop
>> waiting for server to shut down........................................................................................................................... failed
>> pg_ctl: server does not shut down
>> # pg_ctl stop failed: 256
>> # Postmaster PID for node "publisher" is 3679
>> [15:00:38.178](132.032s) Bail out! pg_ctl stop failed
>
> Thanks for reporting this!
>
> From the CI results [1], the failure in 038_walsnd_shutdown_timeout.pl appears
> to occur intermittently on FreeBSD. The failing case tests that, when both
> physical and logical replication are in use with slotsync enabled and both are
> stalled (walreceiver on the standby and the logical apply worker on
> the subscriber are blocked), shutting down the primary completes due to
> wal_sender_shutdown_timeout.
>
> On FreeBSD, however, it seems that after the shutdown request, the physical
> walsender can occasionally keep running, preventing shutdown from completing.
> As a result, pg_ctl stop times out and the test fails.
>
> I’ll investigate the cause. If it takes time to identify, I may temporarily
> disable just this test case so it doesn’t block other development and testing,
> then re-enable it once the issue is fixed.
>
> Regards,
>
> [1]
> https://cirrus-ci.com/build/5134823678803968
> https://cirrus-ci.com/build/5735329598013440
> https://cirrus-ci.com/build/5917696627310592
> https://cirrus-ci.com/build/5742460250357760
>
> --
> Fujii Masao
>
>

I have some CF entries failed on this test case as well, so I tried to look into the problem. I have a finding for your reference.

With a8f45dee917, wal_sender_shutdown_timeout is only enforced while the walsender keeps returning to WalSndCheckShutdownTimeout() in the main loops, but there is a path to enter WalSndDone:
```
/*
* When SIGUSR2 arrives, we send any outstanding logs up to the
* shutdown checkpoint record (i.e., the latest record), wait for
* them to be replicated to the standby, and exit. This may be a
* normal termination at shutdown, or a promotion, the walsender
* is not sure which.
*/
if (got_SIGUSR2)
WalSndDone(send_data);
```

Once entering WalSndDone(), it might call pg_flush() and get stuck:
```
if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;

/* Inform the standby that XLOG streaming is done */
SetQueryCompletion(&qc, CMDTAG_COPY, 0);
EndCommand(&qc, DestRemote, false);
pq_flush();

proc_exit(0);
```

And once stuck, it will never get back to WalSndCheckShutdownTimeout(), so the new GUC timeout cannot rescue it.

In WalSndDoneImmediate(), pq_flush_if_writable() is used, and the comment talks about the possible stuck:
```
/*
* Note that the output buffer may be full during the forced shutdown
* of walsender. If pq_flush() is called at that time, the walsender
* process will be stuck. Therefore, call pq_flush_if_writable()
* instead. Successful reception of the done message with the
* walsender forced into a shutdown is not guaranteed.
*/
pq_flush_if_writable();
```

So, maybe switch to use pq_flush_if_writable() in WalSndDone()?

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/