Re: Updated version of pg_receivexlog

On Thu, Oct 27, 2011 at 7:48 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Thu, Oct 27, 2011 at 12:29, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> On Thu, Oct 27, 2011 at 6:25 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>> On Thu, Oct 27, 2011 at 5:18 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>> Not sure I follow. When we arrive at PQgetCopyData() there should be
>>>> nothing buffered, and if the end of stream happens there it returns
>>>> -1, and we exit, no? So where is the data that's lost?
>>>>
>>>> I do realize we don't actually fsync() and close() in this case - is
>>>> that what you are referring to? But the data should already have been
>>>> write()d, so it should still be there, no?
>>>
>>> Oh, right. Hmm.. xlogdump might be the cause.
>>>
>>> Though I've not read the code of xlogdump, I wonder if it gives up
>>> outputting the contents of WAL file when it finds a partial WAL page...
>>> This strikes me that recovery code has the same problem. No?
>>> IOW, when a partial WAL page is found during recovery, I'm afraid
>>> that page would not be replayed though it contains valid data.
>>
>> My concern was right. When I performed a recovery using the streamed
>> WAL files, the loss of data happened. A partial WAL page was not replayed.
>>
>> To avoid this problem, I think that we should change pg_receivexlog so
>> that it writes WAL data *by the block*, or creates, like walreceiver, WAL file
>> before writing any data. Otherwise, though pg_receivexlog streams WAL
>> data in realtime, the latest WAL data might not be available for recovery.
>
> Ah, so you were recovering data from the last, partial, file? Not from
> a completed file?

Yes. I copied all streamed WAL files to pg_xlog directory and started recovery.

> I'm rewriting the handling of partial files per the other thread
> started by Heikki. The idea is that there will be an actual .partial
> file in there when pg_receivexlog has ended, and you have to deal with
> it manually. The typical way would be to pad it with zeroes to the
> end. Doing such padding would solve this recovery issue, correct?

Yes. But that sounds unuserfriendly. Padding the WAL file manually
is easy-to-do for a user?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center