Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill

From: Ayush Tiwari <ayushtiwari(dot)slg01(at)gmail(dot)com>
To: Alexander Lakhin <exclusion(at)gmail(dot)com>
Cc: Fujii Masao <masao(dot)fujii(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
Date: 2026-11-16 05:26:45
Message-ID: CAJTYsWVpqTOVti3C=B5-yUeQhgJQmV0s1omBhLkmxNzVBW2j2A@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi,

On Sat, 14 Nov 2026 at 19:30, Alexander Lakhin <exclusion(at)gmail(dot)com> wrote:
>
> Hello Fujii-san and Ayush,
>
> 30.10.2026 13:55, Fujii Masao wrote:
> > On Fri, Oct 30, 2026 at 3:59 PM Ayush Tiwari
> > <ayushtiwari(dot)slg01(at)gmail(dot)com> wrote:
> >> Did multiple runs with it, and the changes lgtm.
> > I've pushed the patch. Thanks!
>
> Please look at a new failure of 041_checkpoint_at_promote produced by
> tamandua recently [1]:
> [03:10:10.869](0.025s) ok 4 - psql query died successfully after SIGKILL
> [03:13:15.804](184.935s) # die: timed out waiting for file
>
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log
> contents to match: (?^:all server processes terminated; reinitializing) at
> /home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/
041_checkpoint_at_promote.pl line 166.
> [03:13:15.806](0.003s) 1..4
> timed out waiting for file
>
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log
> contents to match: (?^:all server processes terminated; reinitializing) at
> /home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/
041_checkpoint_at_promote.pl line 166.
> # No postmaster PID for node "master"
>
> 041_checkpoint_at_promote_standby1.log contains:
> 2026-11-12 03:10:10.826 CET [3578531][client backend][6/2:0] LOG:
statement: SELECT pg_backend_pid();
> 2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] LOG: client
backend (PID 3578531) was terminated by signal 9: Killed
> 2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] DETAIL: Failed
process was running: SELECT pg_backend_pid();
> 2026-11-12 03:10:10.862 CET [3577203][postmaster][:0] LOG: terminating
any other active server processes
> 2026-11-12 03:10:10.928 CET [3577203][postmaster][:0] LOG: database
system is ready to accept connections
>
> There is no "terminating any other active server processes" message
> indeed, the server was not shut down.
>
> I managed to reproduce this locally (with the patch attached and with the
> same parallel loop as before) and discovered that this can happen when
> PMState goes from PM_HOT_STANDBY to PM_RUN (via PM_WAIT_BACKENDS) instead
> of PM_HOT_STANDBY -> PM_WAIT_BACKENDS -> PM_WAIT_DEAD_END ->
> PM_NO_CHILDREN -> "all server processes terminated; reinitializing".
>
> This can be explained by two race conditions, with one precondition: the
> startup process doesn't exit before pg_ctl kill.
> 1) The startup process becomes zombie for the postmaster if it exits when
> waitpid() doesn't wait.
> ...
> 2026-11-14 13:29:02.900 UTC [695404][startup][:0] DEBUG: proc_exit(-1): 0
callbacks to make
> ...
> 2026-11-14 13:29:03.088 UTC [695367][postmaster][:0] DEBUG: sending
signal 3/SIGQUIT to startup process with pid 695404
> 2026-11-14 13:29:03.100 UTC [695367][postmaster][:0] LOG:
!!!signal_child| delayed signal 3/SIGQUIT to startup process
> with pid 695404
> ...
> 2026-11-14 13:29:03.170 UTC [695367][postmaster][:0] DEBUG: startup
process 695404 is still running
> ...
> 2026-11-14 13:29:03.198 UTC [695367][postmaster][:0] DEBUG: updating
PMState from PM_WAIT_BACKENDS to PM_RUN
>
>
> 2) SIGQUIT sent to the startup process when it's being executed inside
> exit(0), after the point when signal handlers of a process are not called
> anymore, so SIGQUIT is ignored:
> ...
> 2026-11-14 07:54:44.900 EST [1110043][startup][:0] LOG: !!!proc_exit|
delayed exit(0) for startup process
> ...
> 2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: proc_exit(-1):
0 callbacks to make
> 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG:
!!!signal_child| delayed signal 3/SIGQUIT to startup process
> with pid 1110043
> ...
> 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup
process 1110043 is still running
> ...
> 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid|
ret: 1110043, status: 0
> 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: releasing pm
child slot 89
> 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: updating
PMState from PM_WAIT_BACKENDS to PM_RUN
>
> Corresponding logs for these 2 cases and also a normal run's log are
> attached for reference.

Thanks, I reproduced this with your debugging patch. Looks like a
postmaster bug rather than a problem with the test. Let me try to
write here what I understood.

Normally, startup exits with status 0 after promotion, and the postmaster
takes that as the signal to enter PM_RUN. A backend crash needs the
opposite, stop the other children and rebuild shared memory before
letting the server run again.

Here those two things happen at almost the same time:

1) Startup finishes promotion and is exiting, but the postmaster hasn't
collected its exit status yet.

2) The test kills a backend. The postmaster notices that death first,
sets FatalError and sends SIGQUIT to the remaining children. It recrds
that it told startup to quit by setting STARTUP_SIGNALED.

3) SIGQUIT is too late for startup, it has already exited, or is too far
into exit(0) to handle it? Its exit status is basically still 0.

4) The postmaster then collects that 0 and takes the normal "startup
succeeded" path. It clears FatalError and enters PM_RUN, skipping the
shared-memory reset needed for the backend crash.

That's why the test never sees "all server processes terminated;
reinitializing".

I think we should check STARTUP_SIGNALED even when the exit status is 0.
Something like [2]. With that diff, I tried your patch and reran the tests
multiple
times and it all passed.

Regards,
Ayush

> [1]
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tamandua&dt=2026-11-12%2002%3A04%3A37
[2]

diff --git a/src/backend/postmaster/postmaster.c
b/src/backend/postmaster/postmaster.c
index f53aa9c404d..a2e52b61902 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2317,12 +2317,13 @@ process_pm_child_exit(void)
* during PM_STARTUP due to some dead-end child crashing: in that
* situation, if the startup process dies on the SIGQUIT, we need
* to transition to PM_WAIT_BACKENDS state which will allow
- * PostmasterStateMachine to restart the startup process. (On the
- * other hand, the startup process might complete normally, if we
- * were too late with the SIGQUIT. In that case we'll fall
- * through and commence normal operations.)
+ * PostmasterStateMachine to restart the startup process.
+ *
+ * SIGQUIT may arrive too late to change the startup process's exit
+ * status. If we signaled it, continue the crash restart even if it
+ * exits normally.
*/
- if (!EXIT_STATUS_0(exitstatus))
+ if (!EXIT_STATUS_0(exitstatus) || StartupStatus == STARTUP_SIGNALED)
{
if (StartupStatus == STARTUP_SIGNALED)
{

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Raja Sai pranav 2026-11-16 05:40:30 Re: Re: Re: Assertion failure in WAL summarizer
Previous Message Amit Kapila 2026-11-16 05:00:54 Re: Assertion failure in update_local_synced_slot(), slotsync.c:364