Re: Remove Deprecated Exclusive Backup Mode

Greetings,

* Mark Kirkwood (mark(dot)kirkwood(at)catalyst(dot)net(dot)nz) wrote:
> On 26/02/19 5:41 PM, Stephen Frost wrote:
> >* Mark Kirkwood (mark(dot)kirkwood(at)catalyst(dot)net(dot)nz) wrote:
> >>ISTM that the onus should be on the patch submitter to provide additions to
> >>pg_basebackup that make it as painless as possible for those people *not*
> >>using pgBackRest to continue making backups. Breaking this is just not
> >>right. Submitting patches that mean that people *must* use pgBackRest is
> >>also not right IMHO.
> >I'm sorry that there's some confusion here- to be clear, no one is
> >required to use pgBackRest. pg_basebackup works quite well and wouldn't
> >be impacted by the changes proposed no this thread. The arguments
> >against removing the exclusive backup feature don't have anything to do
> >with pg_basebackup.
>
> Ah yes (checks pg_basbackup code), you are correct! Reading this thread I
> thought I saw a comment to the effect that pg_basebackup was being broken,
> hence the less than impressed post.

No, I don't see breaking pg_basebackup as ok, but I do have concerns
about how it doesn't provide for any validation of the backup unless you
use tar+gzip. Unlike with exclusive mode, at least you don't run the
risk with pg_basebackup that a crash will mean that PG won't come back
up without intervention. I can understand how the comments which you
were responding to might have lead to that confusion.

> Your relentless bashing of people doing their own backups and heavy
> marketing of pgBackRest - unfortunately - made it easy for me to believe
> that this was a possibility that you might see as ok. So - apologies for the
> misunderstanding, however less marketing of your own product would avoid me
> jumping to the wrong conclusion.

As for my personal recommendations, if you go back far enough, you can
see where I used to discuss how to use the exclusive API for doing
backups with people, because maybe they didn't like the tools available
at the time (I didn't, after all), but after seeing how many issues come
from doing that and realizing how bad it is that things like our
documented archive_command doesn't ensure that WAL is actually written
out to disk, I realized why dedicated tools had been written and started
to recommend that people pick one of the dedicated tools instead of
writing their own (and I used to recommend the other solution like
barman right alongside pgBackRest, and I still do make the generic
recommendation at times that people not try to build their own backup
tool) but then after seeing things like CIFS mounts happily throwing
kernel errors while returning success on (obviously, actually failed)
fsync() calls, resulting in corrupted backups that would silently be
corrupt if restored from, I realized that having an actual manifest of
all files backed up and their checksums wasn't a "nice to have" kind of
feature for a backup tool but something critical to a backup solution
(thankfully, David knew that right from the start and so it was always
part of pgBackRest) and so I have a pretty hard time recommending
solutions that don't have that these days.

As for back to the main topic of this thread, do I hope that removing
the exclusive backup mode will cause people to look at and consider
existing, purpose-build, backup solutions for PG, rather than trying to
update their one-off backup shell scripts? Yes, of course I do. Am I
trying to sneak something in that's going to force them to use
pgBackRest? No, of course not. Note that pg_basebackup, barman and
pgBackRest have had support for the non-exclusive API since it was
introduced (or, for barman, even before then if one installed the
pgespresso extension, and pg_basebackup since it was written because it
uses the replication protocol), and so does WAL-G (though I'm not sure
exactly when it got it, or maybe it always had it), and pg_probackup
(which appears to have gotten it in May 2017, less than a year after 9.6
was released), and even the shell-based pitery (which got it at some
point before pg_probackup, it looks like, based on the commits).

Considering there's at least *5* other backup tools which already
support the non-exclusive API, all of which have done so since the
non-exclusive mode was introduced or shortly after, I really have to
question the angle of attack being used here which is claiming that
this thread is motivated by a goal of forcing people to pgBackRest.

The main group who would be impacted by the exclusive mode going away
are those who aren't using an existing solution and who either didn't
get the memo about exclusive mode being deprecated, or decided just to
ignore it, and those are the installations which really do concern me
the most because they're the highest risk for people ending up losing
their data.

Thanks!

Stephen