Re: Timing issue in pg_basebackup

Hello

I can also confirm the race on master and that v3 fixes it. However:

> * Since moving the slot creation, we might run in to trouble if for
> example, --checkpoint=spread is used since the slot is open for longer.

Moving the slot creation in front of BASE_BACKUP causes other new
failures, exactly like the concern you mentioned in the first email,
and I don't think overall it is better than the current issue.

1. The new slot can be invalidated by the backup's own checkpoint

A physical slot starts with the redo pointer of the previous
checkpoint. With the patch it is created, released (so it is
inactive), and stays like that for the whole start-of-backup
checkpoint. At the end of that checkpoint we call
InvalidateObsoleteReplicationSlots(), and our slot is one of the
candidates for invalidation, and can be invalidated based on
max_slot_wal_keep_size or idle_replication_slot_timeout.

This results in backup failures in multiple scenarios, with server log
entries such as

DETAIL: The slot's restart_lsn 0/8B317610 exceeds the limit by 164530672 bytes.

2. The second connection is killed by idle_session_timeout

bgconn is now opened before BASE_BACKUP and sits idle in walsender
command mode until the checkpoint is done. PostgresMain() arms
IDLE_SESSION_TIMEOUT for walsenders too, so with a cluster-wide
idle_session_timeout set and a checkpoint longer than that, the
walsender exits (dropping the temporary slot, and with it the
protection the patch adds), and the backup fails after the checkpoint:

pg_basebackup: error: could not send replication command "IDENTIFY_SYSTEM":
FATAL: terminating connection due to idle-session timeout

(2) is fixable, but (1) is a more tricky problem and I trickier to
solve. Maybe immediately after the slot is created we could use
READ_REPLICATION_SLOT and the START_REPLICATION with the restart_lsn?

Alternatively, instead of reordering, since this is an unlikely race,
maybe early detection and retry could be also an option instead of
reordering the operations? After creating the slot, check restart_lsn:
if it is behond startptr's segment then drop the connection and
restart.