File based Incremental backup v8

From: Marco Nenciarini <marco(dot)nenciarini(at)2ndquadrant(dot)it>
To: Gabriele Bartolini <gabriele(dot)bartolini(at)2ndquadrant(dot)it>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: File based Incremental backup v8
Date: 2015-01-29 14:47:58
Message-ID: 54CA481E.3050405@2ndquadrant.it
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

The current implementation of copydir function is incompatible with LSN
based incremental backups. The problem is that new files are created,
but their blocks are still with the old LSN, so they will not be backed
up because they are looking old enough.

copydir function is used in:

CREATE DATABASE
ALTER DATABASE SET TABLESPACE

I can imagine two possible solutions:

a) wal log the whole copydir operations, setting the lsn accordingly
b) pass to copydir the LSN of the operation which triggered it, and
update the LSN of all the copied blocks

The latter solution is IMO easier to be implemented and does not deviate
much from the current implementation.

I've implemented it and it's attached to this message.

I've also moved the parse_filename_for_notntemp_relation function out of
reinit.c to make it available both to copydir.c and basebackup.c.

I've also limited the LSN comparison to the only MAIN fork, because:

* LSN fork doesn't uses LSN
* VM fork update LSN only when the visibility bit is set
* INIT forks doesn't use LSN. It's only one page anyway.

Regards,
Marco

--
Marco Nenciarini - 2ndQuadrant Italy
PostgreSQL Training, Services and Support
marco(dot)nenciarini(at)2ndQuadrant(dot)it | www.2ndQuadrant.it

Attachment Content-Type Size
0001-public-parse_filename_for_nontemp_relation.patch text/plain 4.8 KB
0002-copydir-LSN.patch text/plain 9.0 KB
0003-File-based-incremental-backup-v8.patch text/plain 47.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Bruce Momjian 2015-01-29 15:13:39 Re: pg_upgrade and rsync
Previous Message Simon Riggs 2015-01-29 14:39:53 Re: GetLockConflicts() and thus recovery conflicts seem pretty broken