Re: Backup solution over unreliable network

On 30/11/18 2:06 μ.μ., Stephen Frost wrote:
> Greetings,
>
> * Achilleas Mantzios (achill(at)matrix(dot)gatewaynet(dot)com) wrote:
>> we've been running our backup solution for the last 5 months to a second
>> site which has an unreliable network connection. We had problems with
>> barman, since it doesn't support backup resume, also no option to disable
>> the replication slot, in the sense, that it is better to sacrifice the
>> backup rather than fill up the primary with WALs and bring the primary down.
>> Another issue is now supporting entirely backing up from the secondary. With
>> barman this is not possible, streaming (or archiving) must originate from
>> the primary.So I want to ask two things here :
>> - Backing up to a remote site over an unreliable channel is a limited use
>> case by itself, it is useful for local PITR restores on specific
>> tables/data, or in case the whole primary suffers a disaster. Is there any
>> other benefit that would justify building a solution for it?
> Please don't build your own solution, it's really quite difficult to get
> backups done correctly.

By "building" I meant setting up, nothing fancier :)

>
>> - I have only read the best reviews about PgBackRest, can PgBackRest address those issues?
> Glad to hear you've read good reviews about pgbackrest. As for
> addressing these issues, pgbackrest has:
>
> - Backup resume
> - Max WAL lag (in other words, you can have it simply start throwing WAL
> away if it can't archive it, rather than allowing the primary to run
> out of disk space)

This is just superb! In our case we had the following architecture (now barman is defunct) :

Primary (consistent snapshots with pg_start/stop_backup)+ --> reliable net (archive_command via rsync) --> WAL repository
   | (async streaming replication)
   | (reliable net)
   V
Standby --> unreliable net (barman via method rsync + barman streaming from standby ***) --> remote cloud provider site(barman)

So Primary and Standby are in the same cloud provider over consistent (mostly) network, whereas the barman (remote recovery) site communicates over internet. We would like to keep the old
functionality (or even add a new PgBackRest node in the main cloud provider, so the question is : is there a way for archive-push to two different stanzas? Or delegate the archive-push to work from
the Standby ?

*** newer barman docs (2.5) say this is not supported (wasn't so clear in 2.4)

> - Backup using the replica, primairly (note that this, currently,
> requires access to the primary, but the bulk of the data comes from
> the replica)
> - Incremental/differential backup
> - Parallel backup/resume and parallel archiving/fetching
> - Backup verification- we checksum every file backed up and verify those
> checksums on a resume, and we make sure that every WAL file needed to
> restore the backup has made it into the archive.
> - Delta restore
>
> Which I believe covers most of the use-cases you brought up.
>
> When we first implemented backup using the replica we had concerns
> regarding doing a 'full' replica-based backup, and we didn't really see
> there being a lot of demand for such a use-case (the replica has access
> to the primary in general if it's a streaming replica, after all...),
> but we might be open to revisiting that.

Thank you a lot! We'll definitely consider PgBackRest.

> Thanks!
>
> Stephen

--
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt