Re: .ready and .done files considered harmful

Hi,

We have addressed the O(n^2) problem which involves directory scan for
archiving individual WAL files by maintaining a WAL counter to identify
the next WAL file in a sequence.

WAL archiver scans the status directory to identify the next WAL file
which needs to be archived. This directory scan can be minimized by
maintaining the log segment number of the current file which is being
archived
and incrementing it by '1' to get the next WAL file in a sequence. Archiver
can check the availability of the next file in status directory and in case
if the
file is not available then it should fall-back to directory scan to get the
oldest
WAL file.

Please find attached patch v1.

Thanks,
Dipesh

On Fri, May 7, 2021 at 1:31 AM Andres Freund <andres(at)anarazel(dot)de> wrote:

> Hi,
>
> On 2021-05-06 21:23:36 +0200, Hannu Krosing wrote:
> > How are you envisioning the shared-memory signaling should work in the
> > original sample case, where the archiver had been failing for half a
> > year ?
>
> If we leave history files and gaps in the .ready sequence aside for a
> second, we really only need an LSN or segment number describing the
> current "archive position". Then we can iterate over the segments
> between the "archive position" and the flush position (which we already
> know). Even if we needed to keep statting .ready/.done files (to handle
> gaps due to archive command mucking around with .ready/done), it'd still
> be a lot cheaper than what we do today. It probably would even still be
> cheaper if we just statted all potentially relevant timeline history
> files all the time to send them first.
>
>
> > Or should we perhaps have a system table for ready-to-archive WAL
> > files to get around limitation sof file system to return just the
> > needed files with ORDER BY ... LIMIT as we already know how to make
> > lookups in database fast ?
>
> Archiving needs to work on a standby so that doesn't seem like an
> option.
>
> Regards,
>
> Andres Freund
>
>
>