Re: parallelizing the archiver

On Sun, Oct 24, 2021 at 2:15 AM Bossart, Nathan <bossartn(at)amazon(dot)com> wrote:
> Here is an attempt at doing this. Archive modules are expected to
> declare _PG_archive_module_init(), which can define GUCs, register
> background workers, etc. This function must at least define the
> archive callbacks. For now, I've introduced two callbacks. The first
> is for checking that the archive module is configured, and the second
> contains the actual archiving logic.

I don't see why this patch should need to make any changes to
internal_load_library(), PostmasterMain(), SubPostmasterMain(), or any
other central point of control, and I don't think it should.
pgarch_archiveXlog() can just load the library at the time it's
needed. That way it only gets loaded in the archiver process, and the
required changes are much more localized. Like instead of asserting
that the functions are initialized, just
load_external_function(libname, "_PG_archive_module_init") and call it
if they aren't.

I think the attempt in check_archive_command()/check_archive_library()
to force exactly one of those two to be set is not going to work well
and should be removed. In general, GUCs whose legal values depend on
the values of other GUCs don't end up working out well. I think what
should happen instead is that if archive_library=shell then
archive_command does whatever it does; otherwise archive_command is
without effect.

--
Robert Haas
EDB: http://www.enterprisedb.com