Re: pg_receivewal fail to streams when the partial file to write is not fully initialized present in the wal receiver directory

On Fri, Apr 22, 2022 at 07:17:37PM +0530, Bharath Rupireddy wrote:
> Right. We find enough disk space and go to write and suddenly the
> write operations fail for some reason or the VM crashes because of a
> reason other than disk space. I think the foolproof solution is to
> figure out the available disk space before prepadding or compressing
> and also use the
> write-first-to-temp-file-and-then-rename-it-to-original-file as
> proposed in the earlier patches in this thread.

Yes, what would count here is only the amount of free space in a
partition. The total amount of space available becomes handy once you
begin introducing things like percentage-based quota policies for the
disk when archiving. The free amount of space could be used to define
a policy based on the maximum number of bytes you need to leave
around, as well, but this is not perfect science as this depends of
what FSes decide to do underneath. There are a couple of designs
possible here. When I had to deal with my upthread case I have chosen
one as I had no need to worry only about Linux, it does not mean that
this is the best choice that would fit with the long-term community
picture. This comes down to how much pg_receivewal should handle
automatically, and how it should handle it.

> Having said that, is there a portable way that we can find out the
> disk space available? I read your response upthread that statvfs isn't
> portable to WIN32 platforms. So, we just say that part of the fix we
> proposed here (checking disk space before prepadding or compressing)
> isn't supported on WIN32 and we just do the temp file thing for WIN32
> alone?

Something like GetDiskFreeSpaceA() would do the trick on WIN32.
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getdiskfreespacea

When it comes to compression, creating a temporary file would only
lead to its deletion, mostly, and extra I/O is never free. Anyway,
why would you need the extra logic of the temporary file at all?
That's basically what the .partial file is as pg_receivewal begins
streaming at the beginning of a segment, to the partial file, each
time it sees fit to restart a streaming cycle.
--
Michael