pgsql: Fix handling of shutdown requests during an early crash restart

Fix handling of shutdown requests during an early crash restart

A smart or fast shutdown during an early crash restart can wait forever
on the checkpointer and I/O workers. FatalError is still set, so
PM_STOP_BACKENDS adds them as processes to wait for, but these are
relaunched by the restart and ignore the SIGTERM that a shutdown sends.
AbortStartTime has been reset at that point, meaning that SIGKILL would
also not fire.

SIGQUIT is now sent instead of SIGTERM when FatalError is set, via
HandleFatalError(), arming AbortStartTime. Going through that routine
rather than signaling the children directly matters for the startup
process: TerminateChildren() marks the startup process as signaled, so
its exit is not later reported as a failure. Note that after sending
the SIGTERM, we check for any running processes, assuming that auxiliary
processes have been already SIGQUIT'd for an immediate shutdown or for a
FatalError.

A test case is added, able to stress the scenario of a shutdown
triggered during the early phases of crash recovery with a custom
restore command waiting for a shutdown by scanning the server logs. The
CI is stable with this test (several runs done); the buildfarm will be
the final judge regarding its stability.

No backpatch is done for now, to be on the conservative side. This
feels also a slightly risky change for v19, which is close to release,
and letting this change bake on HEAD a bit would not hurt.

This problem has been originally reported by Justin Pryzby back in 2023,
without being fixed, and still existed on HEAD.

Reported-by: Justin Pryzby <pryzby(at)telsasoft(dot)com>
Author: Ayush Tiwari <ayushtiwari(dot)slg01(at)gmail(dot)com>
Co-authored-by: Michael Paquier <michael(at)paquier(dot)xyz>
Discussion: https://postgr.es/m/CAJTYsWXCLi00cJ6-_OkMJB4UjyV6EZjc86QwmgE9=iH_C0R-Aw@mail.gmail.com
Discussion: https://postgr.es/m/ZWlrdQarrZvLsgIk@pryzbyj2023

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/12f866cd5f72d65ed4435b8203540efde45ba01d

Modified Files
--------------
src/backend/postmaster/postmaster.c | 14 +++++-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/058_shutdown_crash_restart.pl | 61 +++++++++++++++++++++++
src/test/recovery/t/wait_for_shutdown | 22 ++++++++
4 files changed, 96 insertions(+), 2 deletions(-)