Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails

On Mon, Aug 17, 2026, KUAN-TING KUO wrote:
> (Side note, not 18-specific: the startup process itself dies the same
> way in this state; StartupStatus becomes STARTUP_CRASHED but the
> "shutting down due to startup process failure" exit is only reached
> from PM_NO_CHILDREN, and nothing moves pmState away from PM_STARTUP
> once HandleChildCrash() returns early, so even without io workers the
> postmaster would sit in PM_STARTUP indefinitely with no log entry.
> REL_16/REL_17 have the same shape there.)

Thanks for the report. I think this side note, rather than the io
worker respawn, is the actual bug. I reproduced the silent respawn
loop on Linux master with a hack that makes every child die in
InitPostmasterChild(), and it is unchanged with io_method=sync: the
respawned children are then the checkpointer and the background
writer, but the loop and the silence stay the same. Moving the io
worker respawn out of the reap loop didn't change anything observable
either. (Same with io_method=sync on a stock 18.6 on Windows
following your recipe -- though in my runs the postmaster stayed
responsive to pg_ctl throughout, so I can't speak to the
shutdown-hang part of the report.)

The reason nothing is logged and nothing moves the state machine is
that this state is only reachable while FatalError is set: FatalError
is cleared when WAL redo starts (PMSIGNAL_RECOVERY_STARTED), so a
startup process that crashes during reinitialization before that
point leaves HandleChildCrash() a no-op, as you describe.

This case used to be caught earlier. Until commit 9b43e6793b0f
(affdb2dd5c67 on REL_18_STABLE, first released in 18.4, backpatched
through v15), process_pm_child_exit() had a PM_STARTUP shortcut that
logged "aborting startup due to startup process failure" and exited,
without consulting FatalError. That commit removed the shortcut to
fix a real problem -- the direct exit orphaned the checkpointer and
background writer, which have been running during PM_STARTUP since
v15 -- but the path it falls back to does nothing in the
FatalError-still-set case, so the give-up behavior was lost. On a
build of that commit's parent, the same scenario ends after one
reinitialization cycle with

LOG: startup process (PID 46910) exited with exit code 2
LOG: aborting startup due to startup process failure

and a postmaster exit, as before.

The attached patch restores the give-up while keeping the
orphaned-children fix: if the startup process crashes while
FatalError is still set, log the exit and go through
HandleFatalError(), so the remaining children are signalled and the
existing STARTUP_CRASHED check at PM_NO_CHILDREN terminates the
postmaster. HandleFatalError() loses its Assert(!FatalError) for
that; the repeated call re-signals children launched since the
previous call, which this path wants anyway. Nothing is lost by
exiting: in the wedged state the startup process is never relaunched,
so the server could never have recovered on its own.

With the patch, the reproduction ends about 100 ms after the startup
crash with

LOG: startup process (PID 19613) exited with exit code 2
LOG: aborting startup due to startup process failure
LOG: shutting down due to startup process failure

(12 forks in total, versus 40,000+ in six seconds without the patch;
without the patch these lines never appear). make check passes, and a
SIGKILLed backend on a healthy server still goes through a normal,
logged crash restart and recovers.

Adding Michael and Ayush in CC, as committer and author of
9b43e6793b0f.

Regards,
Zexin Li