Re: parallelizing the archiver

On Fri, Sep 10, 2021 at 6:30 AM Bossart, Nathan <bossartn(at)amazon(dot)com> wrote:
>
> Thanks for chiming in. I'm planning to work on a patch next week.

Great news!

About the technical concerns:

> I'm currently thinking of
> something a bit like autovacuum_max_workers, but the archive workers
> would be created once and would follow a competing consumers model.

In this approach, the launched archiver workers would be kept as long
as the instance is up, or should they be stopped if they're not
required anymore, e.g. if there was a temporary write activity spike?
I think we should make sure that at least one worker is always up.

> Another approach I'm looking at is to use background worker processes,
> although I'm not sure if linking such a critical piece of
> functionality to max_worker_processes is a good idea. However, I do
> see that logical replication uses background workers.

I think that using background workers is a good approach, and the
various guc in that area should allow users to properly configure
archiving too. If that's not the case, it might be an opportunity to
add some new infrastructure that could benefit all bgworkers users.