| From: | Raja Sai pranav <rajasaipranav0(at)gmail(dot)com> |
|---|---|
| To: | pgsql-hackers(at)lists(dot)postgresql(dot)org |
| Cc: | robertmhaas(at)gmail(dot)com |
| Subject: | Incremental backup accepts a prior backup that extends past a timeline switch |
| Date: | 2026-10-23 20:39:09 |
| Message-ID: | CAPHduZC9pWm_vjznmhZNFsDZKLimqc6dKXOozLUKZVPs19zbZg@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi,
While testing incremental backup across failovers, I found a case where
pg_basebackup --incremental accepts a prior backup that is not part of the
server's history, and the combined backup restores without errors but is
corrupt.
The scenario is an ordinary failover with asynchronous replication:
1. A full backup is taken on the primary (timeline 1).
2. Before that backup finishes, the standby stops receiving WAL, so the
backup's WAL range ends after the last WAL the standby replayed.
3. The standby is promoted (timeline 2, switch point inside that range).
4. An incremental backup is taken on the new primary, using the full
backup from step 1 as the prior backup.
Step 4 succeeds. In PrepareForIncrementalBackup(), the end LSN of the prior
backup's final WAL range is only compared with the start of the new backup.
When that timeline is a historical one on this server, the end LSN is not
compared with the switch point in the timeline history, so the check that
already exists for non-final timelines never applies.
The result is that blocks changed on the old primary after the switch point
are not in the new primary's WAL summaries, so the incremental backup omits
them and pg_combinebackup takes them from the prior backup. On PG 18.6, in
my
reproduction, the restored cluster started up fine, but an index scan on the
table's primary key failed with:
ERROR: could not read blocks 412..412 in file "base/5/16390": read
only 0 of 8192 bytes
The new primary's t_pkey has 276 blocks and the old primary's has 553, so
the
combined index kept internal pages from the old history that point past the
end of the 276-block file.
The attached patch adds the missing check: if the prior backup's final
timeline has an end in this server's history, the prior backup's WAL range
must not end after it. The error reuses the wording of the existing
non-final-timeline check:
ERROR: manifest requires WAL from final timeline 1 ending at
0/05631BC0,
but this server switched timelines at 0/04001338
It also adds src/bin/pg_combinebackup/t/013_diverged_parent.pl. The test
creates the condition deterministically by promoting a standby with
recovery_target_lsn set to the prior backup's start LSN, so the switch point
falls inside the prior backup's WAL range. It also checks that an
incremental
relative to a backup taken before the switch still works. Without the code
change, the test fails because the incremental backup succeeds.
Testing on master with the patch:
- pg_combinebackup TAP tests: all 13 pass (124 tests)
- pg_basebackup TAP tests: all 6 pass (324 tests)
- incrementals across a promotion that should keep working (full on the old
primary before the switch, full on the standby before promotion) still
combine and restore with data matching the live server
The same code is in REL_17_STABLE and REL_18_STABLE, so I think this should
be
back-patched to 17.
Regards,
Sai Pranav
| Attachment | Content-Type | Size |
|---|---|---|
| v1-0001-Reject-incremental-backups-whose-prior-backup-out.patch | application/octet-stream | 7.1 KB |
| From | Date | Subject | |
|---|---|---|---|
| Previous Message | Bharath Rupireddy | 2026-10-23 20:28:33 | Re: Parallel vacuum: workers together with leader could overrun cost budget |