[PG19] Promotion hangs behind a backend blocked on a stuck client

Hi,

Since 67c20979ce7, promotion waits until every process absorbs a
ProcSignalBarrier, even with wal_level=replica and no slots. A backend
blocked sending to a client never absorbs it. If a client on the standby
pauses while fetching a result, promotion has no upper bound. A paused
client that stayed connected still held it after 300 seconds. If the
client goes away, promotion waits until TCP gives up, about 16 minutes.
18.6 promotes in 0.1 seconds. This is item 8 in [1].

Manu found why in [2]. ProcessClientWriteInterrupt() only acts on
ProcDiePending, so the barrier stays pending while secure_write() waits.
Once the send buffer is full it hangs every time.

The same wait blocks crash failover and standby
replay when the primary creates its first logical slot. With
synchronous_commit=remote_apply, commits on the primary then wait too.
DROP DATABASE has had the same problem since 15.

The patch absorbs barriers after a blocked wait in secure_read() or
secure_write() wakes up, if interrupts are not held. The normal path
does not change. We may be in the middle of a message, and
AbsorbDataChecksumsBarrier() can throw. So any error is made FATAL and
kept from the client, as Andres suggested in [3]. With a temporary
injection point there, the client sees the connection close. Without
that, the error lands inside a DataRow and the client hangs.

On the REL_19_STABLE head 84572589029 on Linux, all 38 runs with a stuck
client hung on 19, and all finished within 210 ms with the patch. The
attached file has each case and the controls.

DROP DATABASE waits for a barrier too. A stuck client has blocked
it since 15, and 14 is fine. 0001 applies to 15 and later as is, so I
think it should go back to 15.

Not covered is a backend blocked while sending an error message, where
interrupts are held. Andres described it in [3].

[1]
https://postgr.es/m/CAGRkXqSFdjqZP_kwRz7dv6LHX-CSrkDnynNXgntRsCEnQTKxmg@mail.gmail.com
[2] https://postgr.es/m/179191664881.1698521.9947772831570083261@gmail.com
[3]
https://postgr.es/m/CAO6_XqoZb2czT=i=VBQjwBwwo3d21cnMm-EpfYfx0m-O+abFrQ@mail.gmail.com