Logical slot creation/synchronization on a standby may deadlock with recovery conflict resolution

Hi hackers,

While working on eliminating potential deadlock risks of WAIT command, an
agent noticed a similar risk for creating logical replication slots on hot
standby. Here's how it goes:

A statement such as:

SELECT * FROM pg_create_logical_replication_slot('s', 'pgoutput');

retains its executor-registered statement snapshot
while DecodingContextFindStartpoint() searches for a consistent decoding
point. Its WAL reader, read_local_xlog_page_guts(), limits reads to
GetXLogReplayRecPtr().

There seems to be a problematic ordering:

1) The slot-creation statement holds a snapshot and starts waiting for an
additional WAL record to be replayed.

2) The operation in primary generates a cleanup record conflicting with
that snapshot, for example through UPDATE followed by VACUUM while
hot_standby_feedback
is off.
- 3) A running-transactions record needed to complete slot initialization,
which follows the cleanup record.
-
4) Startup process waits for the slot creator's snapshot before replaying
cleanup, while the creator waits for the replay to advance.

This seems serious if max_standby_streaming_delay = -1, since nothing left
to break it automatically. I haven't found a report in the mailing lists
since the debut of this feature. The reason for that could be an
encounterance of it requires a fair coincidence of several factors and the
symptom of being affected is manifested an interval of delay, which could
be masked by the creation of the slot with finite delay setting.

I don't see a clear solution to this potential issue, because the interface
is a function, which means that the held snapshots cannot be popped cleanly
since they belong to the surrounding executor.

Slot synchronization also has a replay-wait path that may participate in
these cycles. I haven't looked into it yet.

Feedbacks are appreciated.

--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.