Re: .ready and .done files considered harmful

Hi,

Thanks for the feedback.

> Which approach do you think we should use? I think we have decent
> patches for both approaches at this point, so perhaps we should see if
> we can get some additional feedback from the community on which one we
> should pursue further.

In my opinion both the approaches have benefits over current implementation.
I think in keep-trying-the-next-file approach we have handled all rare and
specific
scenarios which requires us to force a directory scan to archive the
desired files.
In addition to this with the recent change to force a directory scan at
checkpoint
we can avoid an infinite wait for a file which is still being missed out
despite
handling the special scenarios. It is also more efficient in extreme
scenarios
as discussed in this thread. However, multiple-files-per-readdir approach
is
cleaner with resilience of current implementation.

I agree that we should decide on which approach to pursue further based on
additional feedback from the community.

> The problem I see with this is that pgarch_archiveXlog() might end up
> failing. If it does, we won't retry archiving the file until we do a
> directory scan. I think we could try to avoid forcing a directory
> scan outside of these failure cases and archiver startup, but I'm not
> sure it's really worth it. When pgarch_readyXlog() returns false, it
> most likely means that there are no .ready files present, so I'm not
> sure we are gaining a whole lot by avoiding a directory scan in that
> case. I guess it might help a bit if there are a ton of .done files,
> though.

Yes, I think it will be useful when we have a bunch of .done files and
the frequency of .ready files is such that the archiver goes to wait
state before the next WAL file is ready for archival.

> I agree, but it should probably be something like DEBUG3 instead of
> LOG.

I will update it in the next patch.

Thanks,
Dipesh