Re: finding changed blocks using WAL scanning

On Sat, Apr 20, 2019 at 9:18 AM Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > I think you've got to prevent the WAL from being removed until a
> > .modblock file has been written. In more detail, you should (a) scan
> > all the WAL segments that will be summarized in the .modblock file,
> > (b) write the file under a temporary name, (c) fsync it, (d) rename it
> > into place, (e) fsync it again, and (f) then allow those WAL segments
> > to be removed, if they are otherwise eligible to be removed.
>
> Makes sense. So when you are about to remove WAL, you create the
> .modblock files for all complete WAL files and only create a new one
> when you are about to remove a WAL that was not in a previous .modblock
> file.

There will often be a partial WAL record at the end of each file. So
if you make a .modblock file for WAL files 1-10, you can probably
remove files 1-9, but you probably have to keep WAL file 10 around
until you generate the NEXT .modblock file, because otherwise you
wouldn't be able to read and parse the WAL record that spans the end
of file 10 and the beginning of file 11.

This is a detail that is very very very important to get right.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company