Re: Small fixes needed by high-availability tools

Hi Andrey,

I tested v3 (both patches) on top of master 9f191dce61e, built with
--enable-cassert --enable-debug. Both patches apply in order and build
without warnings. These TAP tests pass:

0001: src/test/recovery t/007_sync_rep.pl (19 tests)
0001+0002: t/007_sync_rep.pl and t/001_stream_rep.pl (90 tests),
test_misc t/003_check_guc.pl

I also ran some manual scenarios with one primary and one synchronous
standby (synchronous_standby_names = 'standby1',
post_recovery_sync_level = remote_apply), the way an HA tool would see
them. What works well:

- With the standby down, a crash restart of the primary rejects normal
connections with the new FATAL, and the connection goes through as
soon as the standby comes back.
- Replication-protocol connections are still accepted, so the standby
can catch up.
- A role with ALTER ROLE ... SET post_recovery_sync_level = off can
connect while everyone else is blocked. That is a nice way to let
the HA agent in.
- Emptying synchronous_standby_names and reloading releases the gate,
and the log says "post-recovery synchronization wait skipped".
- The hot standby, with the same config files, accepts connections
while in recovery.

A few things I noticed:

1. The gate is not limited to crash recovery. After a clean restart
(pg_ctl stop -m fast, then start, with the standby stopped), normal
connections are rejected in the same way. The same happens on a
standby right after promotion. That may well be intended, since
the risk is the same, but the commit message, the test comment and
the docs all say "after crash recovery". Maybe say "after startup
or promotion" so operators are not surprised by a planned restart.

2. pg_isready reports "accepting connections" (exit 0) while every
normal connection is refused. The FATAL uses
ERRCODE_CANNOT_CONNECT_NOW, but it is raised after authentication,
and internal_ping() in fe-connect.c returns PQPING_OK as soon as
auth_req_received is set, before it looks at the SQLSTATE. HA tools
and orchestrators often use pg_isready or PQping as a liveness
check, so in this state they would see a healthy primary that
nobody can use. I am not sure what the best fix is (a mention in
the docs, or a change in libpq), but it seemed worth raising.

3. pg_rewind, using the blocked primary as the source server, fails
with the same FATAL unless it connects as an exempt role. That is
consistent with the design (superusers are not exempt in v3), but
it may be worth a sentence in the docs recommending a role with
post_recovery_sync_level = off for tools such as pg_rewind and
monitoring.

4. With 0002 and synchronous_replication_wait_on_query_cancel = on,
pg_cancel_backend() gives the expected WARNING and the backend keeps
waiting. pg_terminate_backend() still ends the wait, as described.
With statement_timeout = 500ms the client got no message at all
after 4 seconds; it just keeps waiting until the standby list is
changed. That matches the design, but the docs could say explicitly
that statement_timeout is also ignored in this case, since
applications often rely on it as their last safety net.

Thanks for working on this. It is a real problem in HA setups.

Regards,
João Rito

Em qua., 11 de nov. de 2026 às 01:03, Andrey Borodin <x4mmm(at)yandex-team(dot)ru>
escreveu:

> Hi hackers,
>
> I have rebased and reworked the first two patches in this series.
> This is version 3.
>
> I dropped the third patch, which exposed the WAL receiver's write
> position. WAL that has only been written, but not flushed, cannot be
> relied upon when choosing a failover candidate, so that value does not
> provide the durability guarantee needed by an HA tool.
>
> The first attached patch adds post_recovery_sync_level. When set to
> remote_write, on, or remote_apply, a client connection is rejected
> after recovery until the end-of-recovery WAL location reaches the
> configured synchronous standbys. The setting can be assigned per role
> or database, and completion is tracked separately for each level. WAL
> senders are still allowed to connect. An empty
> synchronous_standby_names acts as an explicit operator bypass.
>
> The second patch adds synchronous_replication_wait_on_query_cancel. It
> defaults to off, preserving the current behavior. When it is on,
> query cancellation is reported but does not end a synchronous
> replication wait. ProcDiePending is deliberately unchanged, so
> session termination and postmaster shutdown can still end the wait.
>
> At an Advanced Patch Review session, it was suggested that this policy
> might be better exposed as a hook, allowing an extension to decide
> whether a query cancellation should end the wait. I kept the GUC in
> this version because it provides a simple built-in cluster-wide policy
> without requiring a preload library. A hook could allow more elaborate
> policies, but it would run after local commit while interrupts are held,
> so its API and allowed operations would need to be deliberately narrow.
> I would particularly appreciate opinions on whether this should be a
> GUC, an extension hook, or a combination of both.
>
> Compared with v2, this version:
>
> * separates the startup policy from the query-cancellation policy;
> * stores and checks post-recovery synchronization state under
> SyncRepLock;
> * allows different synchronization levels for roles and databases;
> * does not exempt superusers from the startup gate;
> * keeps the existing behavior by default for both features;
> * documents both settings; and
> * adds TAP coverage for cancellation, startup gating, standby catchup,
> and the explicit empty-configuration bypass.
>
> This problem has now been discussed for almost seven years[0,1], and the
> behavior is in a sensitive part of the commit path. Review from people
> interested in quorum replication and interrupt handling would be
> especially valuable.
>
>
> Best regards, Andrey Borodin.
>
> [0]
> https://www.postgresql.org/message-id/flat/C1F7905E-5DB2-497D-ABCC-E14D4DEE506C%40yandex-team.ru
> [1]
> https://www.postgresql.org/message-id/flat/0B44E464-BA62-4056-9465-3320DD2D0302%40yandex-team.ru
>
>