Re: archive modules

From: Nathan Bossart <nathandbossart(at)gmail(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: "Bossart, Nathan" <bossartn(at)amazon(dot)com>, David Steele <david(at)pgmasters(dot)net>, Fujii Masao <masao(dot)fujii(at)oss(dot)nttdata(dot)com>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>, Michael Paquier <michael(at)paquier(dot)xyz>, Magnus Hagander <magnus(at)hagander(dot)net>, Stephen Frost <sfrost(at)snowman(dot)net>
Subject: Re: archive modules
Date: 2022-02-01 01:36:08
Message-ID: 20220201013608.GA737363@nathanxps13
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Sat, Jan 29, 2022 at 09:01:41PM -0800, Nathan Bossart wrote:
> On Sat, Jan 29, 2022 at 04:31:48PM -0800, Nathan Bossart wrote:
>> On Sat, Jan 29, 2022 at 12:50:18PM -0800, Nathan Bossart wrote:
>>> Here is a new revision. I've moved basic_archive to contrib, hardened it
>>> as suggested, and added shutdown support for archive modules.
>> cfbot was unhappy with v14, so here's another attempt. One other change I
>> am pondering is surrounding pgarch_MainLoop() with PG_TRY/PG_FINALLY so
>> that we can also call the shutdown callback in the event of an ERROR. This
>> might be necessary for an archive module that uses background workers.
> Ugh. Apologies for the noise. cfbot still isn't happy, so here's yet
> another attempt. This new patch set also ensures the shutdown callback is
> called when the archiver process exits.

If basic_archive is to be in contrib, we probably want to avoid restarting
the archiver every time the module ERRORs. I debated trying to add a
generic exception handler that all archive modules could use, but I suspect
many will have unique cleanup requirements. Plus, AFAICT restarting the
archiver isn't terrible, it just causes most of the normal retry logic to
be skipped.

I also looked into rewriting basic_archive to avoid ERRORs and return false
for all failures, but this was rather tedious. Instead, I just introduced
a custom exception handler for basic_archive's archive callback. This
allows us to ERROR as necessary (which helps ensure that failures show up
in the server logs), and the archiver can treat it like a normal failure
and avoid restarting.

Nathan Bossart
Amazon Web Services:

Attachment Content-Type Size
v17-0001-Introduce-archive-modules-infrastructure.patch text/x-diff 12.0 KB
v17-0002-Add-test-archive-module.patch text/x-diff 13.7 KB
v17-0003-Add-documentation-for-archive-modules.patch text/x-diff 30.6 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Nathan Bossart 2022-02-01 01:44:35 Re: Add checkpoint and redo LSN to LogCheckpointEnd log message
Previous Message Andres Freund 2022-02-01 01:35:35 Re: row filtering for logical replication