Re: backup manifests

On 9/19/19 11:00 AM, Robert Haas wrote:

> On Thu, Sep 19, 2019 at 9:51 AM Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> I intend that we should be able to support incremental backups based
>> either on a previous full backup or based on a previous incremental
>> backup. I am not aware of a technical reason why we need to identify
>> the specific backup that must be used. If incremental backup B is
>> taken based on a pre-existing backup A, then I think that B can be
>> restored using either A or *any other backup taken after A and before
>> B*. In the normal case, there probably wouldn't be any such backup,
>> but AFAICS the start-LSNs are a sufficient cross-check that the chosen
>> base backup is legal.
>
> Scratch that: there can be overlapping backups, so you have to
> cross-check both start and stop LSNs.

Overall we have found it's much simpler to label each backup and
cross-check that against the pg version and system id. Start LSN is
pretty unique, but backup labels work really well and are more widely
understood.

>>>> (3) Cross-check a manifest against a backup and complain about extra
>>>> files, missing files, size differences, or checksum mismatches.
>>>
>>> Verification is the best part of the manifest. Plus, you can do
>>> verification pretty cheaply on restore. We also restore pg_control last
>>> so clusters that have a restore error won't start.
>>
>> There's no "restore" operation here, really. A backup taken by
>> pg_basebackup can be "restored" by copying the whole thing, but it can
>> also be used just where it is. If we were going to build something
>> into some in-core tool to copy backups around, this would be a smart
>> way to implement said tool, but I'm not planning on that myself.
>
> Scratch that: incremental backups need a restore tool, so we can use
> this technique there. And it can work for full backups too, because
> why not?

Agreed, once we have a restore tool, use it for everything.

--
-David
david(at)pgmasters(dot)net