Re: Possible race condition in pg_basebackup

Hello Nick

On 2026-Aug-21, Nick Ivanov wrote:

> We are encountering a possible race condition when executing several
> `pg_basebackup --wal-method=stream --slot=... --create-slot` concurrently
> while initialising streaming replicas. Our automation starts 3
> pg_basebackup processes on 3 future replica servers within 1 second of each
> other. One of them almost always fails with "requested WAL segment ... has
> already been removed".

I think this is related to this thread here:
https://www.postgresql.org/message-id/flat/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj%40alibaba-inc.com

and to this commit

Author: Amit Kapila <akapila(at)postgresql(dot)org>
Branch: master Release: REL_19_BR [006dd4b2e] 2025-12-08 05:21:22 +0000
Branch: REL_18_STABLE Release: REL_18_2 [d3ceb2084] 2025-12-08 05:33:14 +0000

Prevent invalidation of newly created replication slots.

A race condition could cause a newly created replication slot to become
invalidated between WAL reservation and a checkpoint.

Previously, if the required WAL was removed, we retried the reservation
process. However, the slot could still be invalidated before the retry if
the WAL was not yet removed but the checkpoint advanced the redo pointer
beyond the slot's intended restart LSN and computed the minimum LSN that
needs to be preserved for the slots.

The fix is to acquire an exclusive lock on ReplicationSlotAllocationLock
during WAL reservation to serialize WAL reservation and checkpoint's
minimum restart_lsn computation. This ensures that, if WAL reservation
occurs first, the checkpoint waits until restart_lsn is updated before
removing WAL. If the checkpoint runs first, subsequent WAL reservations
pick a position at or after the latest checkpoint's redo pointer.

We can't use the same fix for branch 17 and prior because commit
2090edc6f3 changed to compute to the minimum restart_LSN among slot's at
the beginning of checkpoint (or restart point). The fix for 17 and prior
branches is under discussion and will be committed separately.

Reported-by: suyu.cmj <mengjuan(dot)cmj(at)alibaba-inc(dot)com>
Author: Hou Zhijie <houzj(dot)fnst(at)fujitsu(dot)com>
Reviewed-by: Vitaly Davydov <v(dot)davydov(at)postgrespro(dot)ru>
Reviewed-by: Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>
Reviewed-by: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
Backpatch-through: 18
Discussion: https://postgr.es/m/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj@alibaba-inc.com

and to this other commit

Author: Amit Kapila <akapila(at)postgresql(dot)org>
Branch: REL_17_STABLE Release: REL_17_8 [3510ebeb0] 2026-01-08 07:17:56 +0000
Branch: REL_16_STABLE Release: REL_16_12 [24cce33c3] 2026-01-08 07:07:23 +0000
Branch: REL_15_STABLE Release: REL_15_16 [aae05622a] 2026-01-08 06:54:52 +0000
Branch: REL_14_STABLE Release: REL_14_21 [7406df605] 2026-01-08 06:44:28 +0000

Prevent invalidation of newly created replication slots.

A race condition could cause a newly created replication slot to become
invalidated between WAL reservation and a checkpoint.

Previously, if the required WAL was removed, we retried the reservation
process. However, the slot could still be invalidated before the retry if
the WAL was not yet removed but the checkpoint advanced the redo pointer
beyond the slot's intended restart LSN and computed the minimum LSN that
needs to be preserved for the slots.

The fix is to acquire an exclusive lock on ReplicationSlotAllocationLock
during WAL reservation, and a shared lock during the minimum LSN
calculation at checkpoints to serialize the process. This ensures that, if
WAL reservation occurs first, the checkpoint waits until restart_lsn is
updated before calculating the minimum LSN. If the checkpoint runs first,
subsequent WAL reservations pick a position at or after the latest
checkpoint's redo pointer.

We used a similar fix in HEAD (via commit 006dd4b2e5) and 18. The
difference is that in 17 and prior branches we need to additionally handle
the race condition with slot's minimum LSN computation during checkpoints.

Reported-by: suyu.cmj <mengjuan(dot)cmj(at)alibaba-inc(dot)com>
Author: Hou Zhijie <houzj(dot)fnst(at)fujitsu(dot)com>
Author: vignesh C <vignesh21(at)gmail(dot)com>
Reviewed-by: Hayato Kuroda <kuroda(dot)hayato(at)fujitsu(dot)com>
Reviewed-by: Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>
Reviewed-by: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
Backpatch-through: 14
Discussion: https://postgr.es/m/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj@alibaba-inc.com

What version are you using?

If you're using a version that contains these fixes, then we may have
some slightly different bug ...

--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"No hay ausente sin culpa ni presente sin disculpa" (Prov. francés)