pg_combinebackup does not detect missing files

From: David Steele <david(at)pgmasters(dot)net>
To: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: pg_combinebackup does not detect missing files
Date: 2024-04-11 01:36:30
Message-ID: 9badd24d-5bd9-4c35-ba85-4c38a2feb73e@pgmasters.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hackers,

I've been playing around with the incremental backup feature trying to
get a sense of how it can be practically used. One of the first things I
always try is to delete random files and see what happens.

You can delete pretty much anything you want from the most recent
incremental backup (not the manifest) and it will not be detected.

For example:

$ pg_basebackup -c fast -D test/backup/full -F plain
$ pg_basebackup -c fast -D test/backup/incr1 -F plain -i
/home/dev/test/backup/full/backup_manifest

$ rm test/backup/incr1/base/1/INCREMENTAL.2675
$ rm test/backup/incr1/base/1/826
$ /home/dev/test/pg/bin/pg_combinebackup test/backup/full
test/backup/incr1 -o test/backup/combine

$ ls test/backup/incr1/base/1/2675
No such file or directory
$ ls test/backup/incr1/base/1/826
No such file or directory

I can certainly use verify to check the backups individually:

$ /home/dev/test/pg/bin/pg_verifybackup /home/dev/test/backup/incr1
pg_verifybackup: error: "base/1/INCREMENTAL.2675" is present in the
manifest but not on disk
pg_verifybackup: error: "base/1/826" is present in the manifest but not
on disk

But I can't detect this issue if I use verify on the combined backup:

$ /home/dev/test/pg/bin/pg_verifybackup /home/dev/test/backup/combine
backup successfully verified

Maybe the answer here is to update the docs to specify that
pg_verifybackup should be run on all backup directories before
pg_combinebackup is run. Right now that is not at all clear.

In fact the docs say, "pg_combinebackup will attempt to verify that the
backups you specify form a legal backup chain". Which I guess just means
the chain is verified and not the files, but it seems easy to misinterpret.

Overall I think it is an issue that the combine is being driven from the
most recent incremental directory rather than from the manifest.

Regards,
-David

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Hayato Kuroda (Fujitsu) 2024-04-11 01:46:37 RE: Improve eviction algorithm in ReorderBuffer
Previous Message Tom Lane 2024-04-11 01:33:16 Re: Issue with the PRNG used by Postgres