Re: parallelizing the archiver

On Wed, Sep 15, 2021 at 4:14 AM Stephen Frost <sfrost(at)snowman(dot)net> wrote:
>
> > > I'm not proposing to remove existing archive_command. Just deprecate it one-WAL-per-call form.
> >
> > Which is a big API beak.
>
> We definitely need to stop being afraid of this. We completely changed
> around how restores work and made pretty much all of the backup/restore
> tools have to make serious changes when we released v12.

I never said that we should avoid API break at all cost, I said that
if we break the API we should introduce something better. The
proposal to pass multiple file names to the archive command said
nothing about how to tell which ones were successfully archived and
which ones weren't, which is a big problem in my opinion. But I think
we should also consider different approach, such as maintaining some
kind of daemon and asynchronously passing all the WAL file names,
waiting for answers. Or maybe something else. It's just that simply
"passing multiple file names" doesn't seem like a big enough win to
justify an API break to me.

> I definitely don't think that we should be making assumptions that
> changing archive command to start running things in parallel isn't
> *also* an API break too, in any case. It is also a change and there's
> definitely a good chance that it'd break some of the archivers out
> there. If we're going to make a change here, let's make a sensible one.

But doing parallel archiving can and should be controlled with a GUC,
so if your archive_command isn't compatible you can simply just not
use it (on top of having a default of not using parallel archiving, at
least for some times). It doesn't seem like a big problem.