Re: pg_receivewal fail to streams when the partial file to write is not fully initialized present in the wal receiver directory

From: Bharath Rupireddy <bharath(dot)rupireddyforpostgres(at)gmail(dot)com>
To: Michael Paquier <michael(at)paquier(dot)xyz>
Cc: SATYANARAYANA NARLAPURAM <satyanarlapuram(at)gmail(dot)com>, Kyotaro Horiguchi <horikyota(dot)ntt(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: pg_receivewal fail to streams when the partial file to write is not fully initialized present in the wal receiver directory
Date: 2022-04-22 13:47:37
Message-ID: CALj2ACWOwxi8K6j+BG5+Moh6_2MpbZbSijGqGaDKOH_puC5=Ag@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Tue, Apr 19, 2022 at 10:42 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
>
> > I would like to know if there's any problem with the proposed fix.
>
> There is nothing done for the case of compressed segments, meaning
> that you would see the same problem when being in the middle of
> writing a segment compressed with gzip or lz4 in the middle of writing
> it, and that's what you want to avoid here. So the important part is
> not the pre-padding, it is to make sure that there is enough space
> reserved for the handling of a full segment before beginning the work
> on it.

Right. We find enough disk space and go to write and suddenly the
write operations fail for some reason or the VM crashes because of a
reason other than disk space. I think the foolproof solution is to
figure out the available disk space before prepadding or compressing
and also use the
write-first-to-temp-file-and-then-rename-it-to-original-file as
proposed in the earlier patches in this thread.

Having said that, is there a portable way that we can find out the
disk space available? I read your response upthread that statvfs isn't
portable to WIN32 platforms. So, we just say that part of the fix we
proposed here (checking disk space before prepadding or compressing)
isn't supported on WIN32 and we just do the temp file thing for WIN32
alone?

Regards,
Bharath Rupireddy.

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Bharath Rupireddy 2022-04-22 14:02:07 Re: How to generate a WAL record spanning multiple WAL files?
Previous Message Peter Eisentraut 2022-04-22 13:38:35 Re: Dump/Restore of non-default PKs