Re: [PG19] Bugs found comparing 19 with 18, list before I open threads

Hi Shihao,

> 8. Standby promotion waits until every process has accepted a
> ProcSignalBarrier, even with wal_level=replica and no slots. A backend
> blocked sending a result to a client does not accept it. [...] promotion
> hung in 8 of 8 runs on Linux. [...] 18.6 promotes in 0.1 seconds.

I reproduced this on 19rc1 (0b8bff042a9) against 18.6, and tracked down why
it isn't always 8/8.

On promotion the startup process emits a ProcSignalBarrier (generation 1,
from UpdateLogicalDecodingStatusEndOfRecovery), even with wal_level=replica
and no slots. A backend blocked in secure_write() never accepts it: the
latch wakes the write and ProcessClientWriteInterrupt(true) runs, but it only
acts on ProcDiePending, so CHECK_FOR_INTERRUPTS() is never reached. The fix
looks like it belongs in that wait rather than in promotion.

Whether a given stalled backend blocks promotion depends on its local send
buffer, which is why it isn't 8/8 for everyone: unsaturated I got 28/28 hung
pinned to one CPU core but 0/11 on another, and there gdb shows the backend
absorbing the barrier after the kernel freed buffer space. Saturating the
send buffer first -- the state of a client that stalled a while ago -- gives
10/10 hung on 19rc1 and 0/10 on 18.6, on any core.

It is any backend blocked in a socket write, not only a SELECT: COPY ... TO
STDOUT left unread, a stalled pg_basebackup, and a crash failover all hang
too, while idle and idle-in-transaction clients do not. The same wait also
stops plain replay when the first logical slot is created on the primary,
which then blocks synchronous_commit=remote_apply commits there;
max_standby_streaming_delay does not help.

Your 949 s looks like the client's TCP retries running out. With a client
that stays connected but paused (an app thread, a psql in a pager) nothing on
the TCP side ends it: a promotion here was still hung at 450 s, zero-window
persist timer backing off, connection alive. tcp_user_timeout bounds the
wait.

Full write-up with the backtraces and the repro is attached. Happy to help
on the thread when you open it.

Regards,
Manu