Re: Updated backup APIs for non-exclusive backups

Greetings,

On Sun, Nov 25, 2018 at 15:45 Laurenz Albe <laurenz(dot)albe(at)cybertec(dot)at> wrote:

> Stephen Frost wrote:
> > > > On restore, you're
> > > > going to need to create a recovery.conf (at least in released
> versions)
> > > > which provides a restore command (needed even in HEAD today) to get
> the
> > > > old WAL, so having to also create the backup_label file shouldn't be
> > > > that difficult.
> > >
> > > You write "recovery.conf" upon recovery, when you have the restored
> > > backup, so you have it on a file system. No problem adding a file
> then.
> > >
> > > This is entirely different from adding a "backup_label" file to
> > > a backup that has been taken by a backup software in some arbitrary
> > > format in some arbitrary location (think snapshot).
> >
> > There isn’t any need to write the backup label before you restore the
> database,
> > just as you write recovery.conf then.
>
> Granted.
> But it is pretty convenient, and writing it to the data directory right
> away
> is a good thing on top, because it reduces the danger of inadvertedly
> starting the backup without recovery.

Writing it into the data directory is *not* a good thing because as soon as
you do that you risk there being an issue if there’s a crash. Writing into
the backup isn’t a bad idea but if you’re managing your backups then
writing it somewhere else (such as where you write your WAL) and
associating it with the backup (presumably it has a label) should make it
easy to pull back when you restore.

> > > Lastly, if you really want, you can extract out the data from
> > > > pg_stop_backup in whatever your post-backup script is.
> > >
> > > Come on, now.
> > > You usually use backup techniques like that because you can't get
> > > your large database backed up in the available time window otherwise.
> >
> > I’m not following what you’re trying to get at here, why can’t you
> extract
> > the data for the backup label from pg_stop_backup..? Certainly other
> tools
> > do, even ones that do extremely fast parallel backups.. the two are
> > completely independent.
> >
> > Did you think I meant pg_basebackup..? I certaily didn’t.
>
> Oh yes, I misunderstood. Sorry.
>
> Yes, you can come up with a post-backup script that somehow communicates
> with your pre-backup script to get the information, but it sure is
> inconvenient. Simplicity is good in backup solutions, because complicated
> things tend to break more easily.

Not sure what communication is necessary here..? The data needed for the
backup label file comes from pg_stop_backup in a non-exclusive backup. You
*should* be grabbing the starting WAL from the start backup as well, to
confirm that you have all of the WAL for the backup, but you don’t actually
need to in order to write out the backup label.

> > I thought our goal is to provide convenient backup methods...
> >
> > Correctness would be first and having a broken system because of a crash
> during a backup isn’t correct.
>
> "Not starting up without manual intervention" is not actually broken...

Of course it is. Having the behavior of the system be completely different
depending on if a backup happened to be running or not is just plain bad
and is broken. It’s not a feature. Had this issue been realized when the
exclusive backup mode was developed I suspect it never would have been
implemented that way to begin with...

Thanks!

Stephen