Re: .ready and .done files considered harmful

Hi,

On 2021-05-06 21:23:36 +0200, Hannu Krosing wrote:
> How are you envisioning the shared-memory signaling should work in the
> original sample case, where the archiver had been failing for half a
> year ?

If we leave history files and gaps in the .ready sequence aside for a
second, we really only need an LSN or segment number describing the
current "archive position". Then we can iterate over the segments
between the "archive position" and the flush position (which we already
know). Even if we needed to keep statting .ready/.done files (to handle
gaps due to archive command mucking around with .ready/done), it'd still
be a lot cheaper than what we do today. It probably would even still be
cheaper if we just statted all potentially relevant timeline history
files all the time to send them first.

> Or should we perhaps have a system table for ready-to-archive WAL
> files to get around limitation sof file system to return just the
> needed files with ORDER BY ... LIMIT as we already know how to make
> lookups in database fast ?

Archiving needs to work on a standby so that doesn't seem like an
option.

Regards,

Andres Freund