Re: Postmaster fails to shut down right after crash restart

> On 24 Apr 2025, at 17:06, Sergey Shinderuk <s(dot)shinderuk(at)postgrespro(dot)ru> wrote:
>
> the postmaster could fail to shut down right after crash restart. It could get stuck in the PM_WAIT_BACKENDS state forever.

Hi Sergey,

I looked at this and reproduced it on current master. Your diagnosis looks
correct: if a smart or fast shutdown request arrives during crash-restart
reinitialization, before PMSIGNAL_RECOVERY_STARTED, FatalError remains
set. The normal shutdown path then sends SIGTERM to the processes started
for crash recovery and waits in PM_WAIT_BACKENDS, but the checkpointer
ignores SIGTERM.

PFA a three-patch series:

1. Allow wait injection points to be controlled through files under
pg_injection_points/ in the data directory.
2. Add a deterministic test for this bug, covering both smart and fast
shutdown.
3. Fix the shutdown path.

Michael, this looks like the concrete test case that was missing from
our July discussion [0]. The test needs all three operations you
listed without relying on SQL: request attachment before triggering
crash restart so that the point is installed in the replacement shared
memory, observe that the startup process has reached it, and release
that process after the postmaster has handled the shutdown request.

Patch 0001 implements the filesystem approach you suggested. Creating

pg_injection_points/<point>/

causes the wait point to be attached when shared memory is initialized.
A process waiting there publishes

pg_injection_points/<point>/<pid>

and removing that file releases the process. The directories are
rescanned when shared memory is recreated after a crash, so the wait
point survives the exact reset that makes the SQL interface unusable
here. Wait points attached only through SQL do not create filesystem
state.

With patches 0001 and 0002, the test reproduces the hang. Patch 0003
handles a shutdown request received with FatalError set by using
HandleFatalError(PMQUIT_FOR_STOP, false). That sends the crash-recovery
children SIGQUIT instead of waiting for them after SIGTERM. There is no
graceful shutdown checkpoint to perform in the middle of crash recovery;
the next startup will run recovery again.

The filesystem interface for wait injection points is intended for HEAD.
The fix itself is independent and looks suitable for backpatching.

Thoughts?

Thank you!

Best regards, Andrey Borodin.

[0]
https://www.postgresql.org/message-id/flat/aher0VsjJ8xeNgLq%40paquier.xyz