Re: Would it be possible to have parallel archiving?

From: Stephen Frost <sfrost(at)snowman(dot)net>
To: Andrey Borodin <x4mmm(at)yandex-team(dot)ru>
Cc: David Steele <david(at)pgmasters(dot)net>, hubert depesz lubaczewski <depesz(at)depesz(dot)com>, pgsql-hackers mailing list <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Would it be possible to have parallel archiving?
Date: 2018-08-28 20:07:54
Message-ID: 20180828200754.GI3326@tamriel.snowman.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Greetings,

* Andrey Borodin (x4mmm(at)yandex-team(dot)ru) wrote:
> > 28 авг. 2018 г., в 14:08, Stephen Frost <sfrost(at)snowman(dot)net> написал(а):
> > * David Steele (david(at)pgmasters(dot)net <mailto:david(at)pgmasters(dot)net>) wrote:
> >> On 8/28/18 8:32 AM, Stephen Frost wrote:
> >> To be clear, pgBackRest uses the .ready files in archive_status to
> >> parallelize archiving but still notifies PostgreSQL of completion via
> >> the archive_command mechanism. We do not modify .ready files to .done
> >> directly.
> >
> > Right, we don't recommend mucking around with that directory of files.
> > Even if that works today (which you'd need to test extensively...),
> > there's no guarantee that it'll work and do what you want in the
> > future...
> WAL-G modifies archive_status files.

Frankly, I've heard far too many concerns and issues with WAL-G to
consider anything it does at all sensible.

> This path was chosen to limit state preserved between WAL-G runs (archiving to S3) and further push archiving performance.

I still don't think it's a good idea and I specifically recommend
against making changes to the archive status files- those are clearly
owned and managed by PG and should not be whacked around by external
processes.

> Indeed, it was very hard to test. Also, this makes impossible to use two archiving system simultaneously for transit period.

The testing in WAL-G seems to be rather lacking from what I've seen.

It's not clear to me what you're suggesting wrt 'two archiving system
simultaneously for transit period', but certainly whacking the archive
status files around doesn't make it easier to have multiple archive
commands trying to simultaneously archive WAL files.

Thanks!

Stephen

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andrey Borodin 2018-08-28 20:34:34 Re: Would it be possible to have parallel archiving?
Previous Message Andrey Borodin 2018-08-28 20:01:21 Re: Would it be possible to have parallel archiving?