Re: finding changed blocks using WAL scanning

On Mon, Apr 22, 2019 at 08:52:11PM -0400, Bruce Momjian wrote:
> Well, the interesting question is whether the server will generate a
> single modblock file for all WAL in pg_wal only right before we are
> ready to expire some WAL, or whether modblock files will be generated
> offline, perhaps independent of the server, and perhaps by aggregating
> smaller modblock files.
>
> To throw out an idea, what if we had an executable that could generate a
> modblock file by scanning a set of WAL files? How far would that take
> us to meeing incremental backup needs? I can imagine db/relfilenode oid
> volatility could be a problem, but might be fixable.

Well, this actually brings up a bunch of questions:

* How often do we create blockmod files? Per segment, per checkpoint,
at WAL deletion time (1GB?)

* What is the blockmod file format? dboid, relfilenode, blocknum?
Use compression? Sorted?

* How do we create incremental backups?

* What is the incremental backup file format?

* How do we apply incremental backups to base backups?

And there are some secondary questions:

* Can blockmod files be merged?

* Can incremental backups be merged?

* Can blockmod files be used for restore prefetching?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +