pgsql: Fix crash-restart race in 041_checkpoint_at_promote.pl

Fix crash-restart race in 041_checkpoint_at_promote.pl

The buildfarm member bushmaster reported failures in the final test query
in 041_checkpoint_at_promote.pl with "the database system is in recovery
mode".

After killing a backend to trigger crash recovery, the test should wait
for the postmaster to terminate the remaining server processes and restart
the server, then wait for recovery to finish and connections to be accepted
before running the final query.

Previously, the test assumed that the killed backend's connection closing
meant the postmaster had noticed the crash and begun restarting. However,
the connection can close before the postmaster notices the crash. The
subsequent connection check can then succeed before recovery starts,
causing the test to mistake this for recovery having completed. The final
query could then run during recovery and fail.

Fix this by waiting for the postmaster to log that all server processes
have terminated and it is reinitializing before checking that connections
are accepted again. This ensures that the postmaster has noticed the crash
and begun restarting before the test waits for recovery to finish.

Backpatch to v17, where this test was introduced.

Reported-by: Alexander Lakhin <exclusion(at)gmail(dot)com>
Author: Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
Reviewed-by: Ayush Tiwari <ayushtiwari(dot)slg01(at)gmail(dot)com>
Discussion: https://postgr.es/m/e7abd282-655d-495e-94c9-e38a58917ebf@gmail.com
Backpatch-through: 17

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/eda9d98fac4f22d31ffd3507ca7b470da572a1c3

Modified Files
--------------
src/test/recovery/t/041_checkpoint_at_promote.pl | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)