Possible race condition in pg_basebackup

Hello,

We are encountering a possible race condition when executing several
`pg_basebackup --wal-method=stream --slot=... --create-slot` concurrently
while initialising streaming replicas. Our automation starts 3
pg_basebackup processes on 3 future replica servers within 1 second of each
other. One of them almost always fails with "requested WAL segment ... has
already been removed".

wal_keep_size = 0 (the default) on the server.

The server log shows three checkpoints starting within 1 second of each
other, recycling the WAL segment needed by the failing pg_basebackup.

Looking at the pg_basebackup code, we can see that it sends the BASE_BACKUP
command to the server, triggering the checkpoint, before
calling StartLogStreamer() that creates the slot and begins streaming WAL.
The time gap between the BASE_BACKUP and the slot creation seems to be the
reason for the failure we see.

Two obvious workarounds exist: create slots as a separate step before
running pg_basebackup, or update wal_keep_size to some sensible value to
prevent WAL segments from being recycled prematurely.

It seems to me, however, that pg_basebackup can be improved, regardless of
the existing workarounds, to create the slot, if it's being requested,
before starting the actual backup.

I'd like to hear feedback from the community on this.

Thanks

Nick Ivanov
EDB