Re: trying again to get incremental backup

From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>
Cc: Dilip Kumar <dilipbalaut(at)gmail(dot)com>, Andres Freund <andres(at)anarazel(dot)de>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: trying again to get incremental backup
Date: 2023-11-20 15:42:47
Message-ID: CA+TgmobY+snP5=b_J92sSaSTYKTCTTpHmK0Q=e1CLD-UxkGktA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Fri, Nov 17, 2023 at 5:01 AM Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org> wrote:
> I made a pass over pg_combinebackup for NLS. I propose the attached
> patch.

This doesn't quite compile for me so I changed a few things and
incorporated it. Hopefully I didn't mess anything up.

Here's v11. In addition to incorporating Álvaro's NLS changes, with
the off-list help of Jakub Wartak, I finally tracked down two one-line
bugs in BlockRefTableEntryGetBlocks that have been causing the cfbot
to blow up on these patches. What I hadn't realized is that cfbot runs
with the relation segment size changed to 6 blocks, which tickled some
code paths that I wasn't exercising locally. Thanks a ton to Jakub for
the help running this down. cfbot was unhappy about a %lu so I've
changed that to %zu in this version, too. Finally, the previous
version of this patch set had some pgindent damage, so that is
hopefully now cleaned up as well.

I wish I had better ideas about how to thoroughly test this. I've got
a bunch of different tests for pg_combinebackup and I think those are
good, but the bugs mentioned in the previous paragraph show that those
aren't sufficient to catch all of the logic errors that can exist,
which is not great. But, as I say, I'm not quite sure how to do
better, so I guess I'll just need to keep fixing problems as we find
them.

--
Robert Haas
EDB: http://www.enterprisedb.com

Attachment Content-Type Size
v11-0003-Move-src-bin-pg_verifybackup-parse_manifest.c-in.patch application/octet-stream 4.3 KB
v11-0002-Rename-pg_verifybackup-s-JsonManifestParseContex.patch application/octet-stream 3.8 KB
v11-0001-Rename-JsonManifestParseContext-callbacks.patch application/octet-stream 5.3 KB
v11-0005-Add-support-for-incremental-backup.patch application/octet-stream 216.7 KB
v11-0004-Add-a-new-WAL-summarizer-process.patch application/octet-stream 135.0 KB
v11-0006-Add-new-pg_walsummary-tool.patch application/octet-stream 17.8 KB
v11-0007-Test-patch-Enable-summarize_wal-by-default.patch application/octet-stream 4.7 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Robert Haas 2023-11-20 16:11:13 Re: Add recovery to pg_control and remove backup_label
Previous Message Nazir Bilal Yavuz 2023-11-20 14:43:17 Re: Show WAL write and fsync stats in pg_stat_io