Re: Updated version of pg_receivexlog

On Thu, Oct 27, 2011 at 6:25 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Thu, Oct 27, 2011 at 5:18 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> Not sure I follow. When we arrive at PQgetCopyData() there should be
>> nothing buffered, and if the end of stream happens there it returns
>> -1, and we exit, no? So where is the data that's lost?
>>
>> I do realize we don't actually fsync() and close() in this case - is
>> that what you are referring to? But the data should already have been
>> write()d, so it should still be there, no?
>
> Oh, right. Hmm.. xlogdump might be the cause.
>
> Though I've not read the code of xlogdump, I wonder if it gives up
> outputting the contents of WAL file when it finds a partial WAL page...
> This strikes me that recovery code has the same problem. No?
> IOW, when a partial WAL page is found during recovery, I'm afraid
> that page would not be replayed though it contains valid data.

My concern was right. When I performed a recovery using the streamed
WAL files, the loss of data happened. A partial WAL page was not replayed.

To avoid this problem, I think that we should change pg_receivexlog so
that it writes WAL data *by the block*, or creates, like walreceiver, WAL file
before writing any data. Otherwise, though pg_receivexlog streams WAL
data in realtime, the latest WAL data might not be available for recovery.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center