BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails

The following bug has been logged on the website:

Bug reference: 19623
Logged by: KUAN-TING KUO
Email address: m0935388420(at)gmail(dot)com
PostgreSQL version: 18.4
Operating system: Windows 11 Pro for Workstations 10.0.26200 (x64)
Description:

PostgreSQL version: 18.4 ("PostgreSQL 18.4 on x86_64-windows, compiled by
msvc-19.44.35228, 64-bit"; conda-forge build)
Operating system: Windows 11 Pro for Workstations 10.0.26200 (x64)
Configuration: initdb defaults except port/listen_addresses; io_method
= worker (default), io_workers = 3 (default)
Note on version: tested on 18.4 (the build I have); the code path
described below is
unchanged in the REL_18_6 tag / REL_18_STABLE
(postmaster.c, the io worker
branch of process_pm_child_exit(), lines 2504-2512 in
REL_18_6).

Summary
-------
If, after a crash restart ("all server processes terminated;
reinitializing"),
the postmaster's freshly spawned children die immediately at process start,
the postmaster on 18.4 goes into a hot loop that:

* respawns io workers ~3 times per second, forever;
* writes nothing further to the server log;
* keeps the listen socket open (postmaster.pid still says "ready",
pg_ctl status says "server is running"), while every connection is
reset (pg_isready exit 2);
* dispatches signals only every couple of minutes, so `pg_ctl reload`
took 134 s to be honoured and `pg_ctl stop -m fast` / `-m immediate`
with timeouts of 10-240 s reported "server does not shut down";
only TerminateProcess ended it.

The condition that made every child die is Windows-specific (see
"How I hit it"), but the postmaster behaviour it exposes looks like a
generic
18.x issue in process_pm_child_exit(): the reap loop calls
maybe_adjust_io_workers() synchronously for every dead io worker, so as long
as a replacement worker dies faster than the next CreateProcess()/fork()
completes, the `while ((pid = waitpid(-1, ...)) > 0)` loop never drains and
control never returns to ServerLoop() (no WaitEventSetWait -> no signal
dispatch, no LaunchMissingBackgroundProcesses, no shutdown handling).
HandleChildCrash() returns early because FatalError is still set from the
first crash, so none of these deaths is logged.

How I hit it (Windows)
----------------------
The postmaster had been started with `pg_ctl -w start` from a cmd.exe that
owned a (hidden) console. That console's conhost.exe was later killed
together
with the cmd.exe (a `taskkill /T /F` on the cmd process tree; conhost.exe is
a
child of the console-owning process, the postmaster is not, so the
postmaster
survived attached to a console whose server is gone;
AttachConsole(postmaster)
from another process fails with error 233 ERROR_PIPE_NOT_CONNECTED). From
then
on every process the postmaster creates with CreateProcess() inherits that
dead
console and dies during process initialisation with exit status 0xC0000142
(STATUS_DLL_INIT_FAILED). The postmaster itself keeps running and, until a
child needs to be spawned, still works (a SIGHUP sent in this state was
logged
immediately).

Reproduction (18.4, Windows; deterministic, done 4/4 times on a fresh
initdb)
------------------------------------------------------------------------------
1. initdb -D data -U postgres -A trust --no-locale -E UTF8; set port = 5499,
listen_addresses = '127.0.0.1'.
2. From Python, start a hidden console whose cmd.exe runs
`pg_ctl -D data -l server.log -w start` and then lingers:
subprocess.Popen(["cmd.exe", "/c", "owner.bat"],
creationflags=CREATE_NEW_CONSOLE,
startupinfo=<SW_HIDE>)
Wait for pg_isready = 0.
3. `taskkill /T /F /PID <that cmd.exe pid>` -> kills cmd.exe and its child
conhost.exe; postmaster survives (verify: AttachConsole(pid) -> 233).
4. `psql -h 127.0.0.1 -p 5499 -U postgres -c "select 1"` ->
"server closed the connection unexpectedly". server.log:
LOG: client backend (PID 52868) was terminated by exception
0xC0000142
HINT: See C include file "ntstatus.h" for a description of the
hexadecimal value.
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
and nothing after that.
5. Observe (numbers from one run, all runs alike):
- postmaster main thread ~80-100 % of one core; sampled 20x with a
GetThreadContext-based sampler: every sample inside
KERNELBASE!CreateProcessInternalW (NtCreateUserProcess /
BasepQueryAppCompat / CsrClientCallServer), never in a wait.
- 72 distinct child postgres.exe processes appeared in 20 s; opening each
one and waiting: all 72 exited with 0xC0000142. While still suspended,
each child already had the postmaster's 150 MB shared-memory range
reserved (VirtualQueryEx: RESERVE/PRIVATE at the same base), i.e.
pgwin32_ReserveSharedMemoryRegion() succeeded and the child was
resumed;
this is not the ASLR/487 retry loop.
- `pg_ctl reload` (SIGHUP) sent right after step 4 was logged
("received SIGHUP, reloading configuration files") only 134 s later.
- `pg_ctl stop -m immediate -t 240` -> "server does not shut down" after
261 s; server.log grew by nothing but that one SIGHUP line.
- port still LISTENING, postmaster.pid still present with status "ready",
pg_isready -> 2 throughout.

Where I think the loop is (src/backend/postmaster/postmaster.c,
REL_18_STABLE)
------------------------------------------------------------------------------
process_pm_child_exit():

while ((pid = waitpid(-1, &exitstatus, WNOHANG)) > 0)
{
...
/* Was it an IO worker? */
if (maybe_reap_io_worker(pid))
{
if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
HandleChildCrash(pid, exitstatus, _("io worker"));

maybe_adjust_io_workers(); <-- spawns a replacement
here
continue;
}

maybe_adjust_io_workers() spawns synchronously (`while (io_worker_count <
io_workers) StartChildProcess(B_IO_WORKER)`). On this machine one
CreateProcess() of postgres.exe takes ~300-450 ms while a child that fails
initialisation is dead a few ms after ResumeThread(). So by the time the
freshly spawned worker's launch returns, the previously spawned worker has
already died and its exit is sitting in the win32 waitpid() completion
queue:
the while loop finds another dead io worker, spawns another replacement, and
so
on. The loop only exits on the rare occasion that a death has not been
queued
yet when waitpid() polls, which is why signals were serviced roughly every
two
minutes rather than never.

HandleChildCrash() (same file) begins with

if (FatalError || Shutdown == ImmediateShutdown)
return;

and FatalError is only cleared when the startup process completes, so every
death after "reinitializing" is silent. (Side note, not 18-specific: the
startup process itself dies the same way in this state; StartupStatus
becomes
STARTUP_CRASHED but the "shutting down due to startup process failure" exit
is
only reached from PM_NO_CHILDREN, and nothing moves pmState away from
PM_STARTUP once HandleChildCrash() returns early, so even without io workers
the postmaster would sit in PM_STARTUP indefinitely with no log entry.
REL_16/REL_17 have the same shape there.)

Expected behaviour
------------------
Either the postmaster should give up (as the comment above the
STARTUP_CRASHED
check says: "we don't try to reinitialize when the startup process fails,
because more than likely it will just fail again and we will keep trying
forever"), or at least: repeated child deaths after a crash restart should
be
logged, replacement io workers should not be spawned from inside the reap
loop
(deferring to LaunchMissingBackgroundProcesses() would let signal handling
and
shutdown requests run between attempts), and some backoff/limit should apply
so a persistently failing child kind cannot monopolise the postmaster.

I can rerun the reproduction with additional instrumentation or provide a
minidump of the spinning postmaster on request.