| From: | Magnus Hagander <magnus(at)hagander(dot)net> |
|---|---|
| To: | Zsolt Parragi <zsolt(dot)parragi(at)percona(dot)com> |
| Cc: | Jesper St John <jesper(at)stjohn(dot)se>, pgsql-bugs(at)lists(dot)postgresql(dot)org, Kirill Reshke <reshkekirill(at)gmail(dot)com> |
| Subject: | Re: Timing issue in pg_basebackup |
| Date: | 2026-11-16 10:26:14 |
| Message-ID: | CABUevEyyZqej1VS=JNhCgXJQP+2-VHYQccnXz079k=7c_7Wqsw@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-bugs |
On Thu, 12 Nov 2026 at 22:19, Zsolt Parragi <zsolt(dot)parragi(at)percona(dot)com>
wrote:
> Hello
>
> I can also confirm the race on master and that v3 fixes it. However:
>
> * Since moving the slot creation, we might run in to trouble if for
> > example, --checkpoint=spread is used since the slot is open for longer.
>
> Moving the slot creation in front of BASE_BACKUP causes other new
> failures, exactly like the concern you mentioned in the first email,
> and I don't think overall it is better than the current issue.
>
> 1. The new slot can be invalidated by the backup's own checkpoint
>
> A physical slot starts with the redo pointer of the previous
> checkpoint. With the patch it is created, released (so it is
> inactive), and stays like that for the whole start-of-backup
> checkpoint. At the end of that checkpoint we call
> InvalidateObsoleteReplicationSlots(), and our slot is one of the
> candidates for invalidation, and can be invalidated based on
> max_slot_wal_keep_size or idle_replication_slot_timeout.
>
> This results in backup failures in multiple scenarios, with server log
> entries such as
>
> DETAIL: The slot's restart_lsn 0/8B317610 exceeds the limit by 164530672
> bytes.
> 2. The second connection is killed by idle_session_timeout
>
> bgconn is now opened before BASE_BACKUP and sits idle in walsender
> command mode until the checkpoint is done. PostgresMain() arms
> IDLE_SESSION_TIMEOUT for walsenders too, so with a cluster-wide
> idle_session_timeout set and a checkpoint longer than that, the
> walsender exits (dropping the temporary slot, and with it the
> protection the patch adds), and the backup fails after the checkpoint:
>
> pg_basebackup: error: could not send replication command "IDENTIFY_SYSTEM":
> FATAL: terminating connection due to idle-session timeout
>
> (2) is fixable, but (1) is a more tricky problem and I trickier to
> solve. Maybe immediately after the slot is created we could use
> READ_REPLICATION_SLOT and the START_REPLICATION with the restart_lsn?
>
> Alternatively, instead of reordering, since this is an unlikely race,
> maybe early detection and retry could be also an option instead of
> reordering the operations? After creating the slot, check restart_lsn:
> if it is behond startptr's segment then drop the connection and
> restart.
>
Per some off-list discussion with Jesper. I don't think that wil work, as
READ_REPLICATION_SLOT won't actually advance the last use time on it, so it
would still time out.
I think the correct thing to do here is to start the full log streamer
*before* starting the base backup. This will lead to including just a
little bit unnecessarily much WAL in the backup, but it won't be much - we
already need to keep the WAL back to the start of the base backup
checkpoint start. So the only extra WAL to be included would be between the
log streaming started and the base backup command running, which would not
be much. The only real difference would be the streaming starts already
during the checkpoint, so with a slow cheakpoing it won't start the
streamer with a large backlog.
--
Magnus Hagander
Me: https://www.hagander.net/ <http://www.hagander.net/>
Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Aleksander Alekseev | 2026-11-16 10:45:34 | Re: BUG #19772: ERROR: corrupt MVNDistinct entry for GROUP BY c0, ROLLUP (abs(c0)) with ndistinct statistics |
| Previous Message | Hayato Kuroda (Fujitsu) | 2026-11-16 08:01:04 | RE: BUG #19892: Logical replication skips UPDATE/DELETE of rows with NULL in a hash-indexed subscriber column |