Re: Extending BASE_BACKUP in replication protocol: incremental backup and backup format

Hi,

On 2014-01-14 21:47:43 +0900, Michael Paquier wrote:
> I would like to propose the following things to extend BASE_BACKUP to
> retrieve a backup from a stream:
> - Addition of an option FORMAT, to control the output format of
> backup, with possible options as 'plain' and 'tar'. Default is tar for
> backward compatibility purposes. The purpose of this option is to make
> easier for backup tools playing with postgres to retrieve and backup
> and analyze it on the fly, the purpose being to filter and analyze the
> data while it is being received without all the tar decoding
> necessary, what would consist in copying portions of pg_basebackup
> code more or less.

We'd need our own serialization format since we're dealing with more
than one file, what would be the point?

> - Addition of an option called INCREMENTAL to send an incremental
> backup to the client. This option uses as input an LSN, and sends back
> to client relation pages (in the shape of reduced relation files) that
> are newer than the LSN specified by looking at pd_lsn of
> PageHeaderData. In this case the LSN needs to be determined by client
> based on the latest full backup taken. This option is particularly
> interesting to reduce the amount of data taken between two backups,
> even if it increases the restore time as client needs to reconstitute
> a base backup depending on the recovery target and the pages modified.
> Client would be in charge of rebuilding pages from incremental backup
> by scanning all the blocks that need to be updated based on the full
> backup as the LSN from which incremental backup is taken is known. But
> this is not really something the server cares about... Such things are
> actually done by pg_rman as well.

Why not just rely on WAL replay since you're relying on the consistency
of the standby anyway?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services