Re: Updated version of pg_receivexlog

On Thu, Oct 27, 2011 at 12:29, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Thu, Oct 27, 2011 at 6:25 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> On Thu, Oct 27, 2011 at 5:18 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>> Not sure I follow. When we arrive at PQgetCopyData() there should be
>>> nothing buffered, and if the end of stream happens there it returns
>>> -1, and we exit, no? So where is the data that's lost?
>>>
>>> I do realize we don't actually fsync() and close() in this case - is
>>> that what you are referring to? But the data should already have been
>>> write()d, so it should still be there, no?
>>
>> Oh, right. Hmm.. xlogdump might be the cause.
>>
>> Though I've not read the code of xlogdump, I wonder if it gives up
>> outputting the contents of WAL file when it finds a partial WAL page...
>> This strikes me that recovery code has the same problem. No?
>> IOW, when a partial WAL page is found during recovery, I'm afraid
>> that page would not be replayed though it contains valid data.
>
> My concern was right. When I performed a recovery using the streamed
> WAL files, the loss of data happened. A partial WAL page was not replayed.
>
> To avoid this problem, I think that we should change pg_receivexlog so
> that it writes WAL data *by the block*, or creates, like walreceiver, WAL file
> before writing any data. Otherwise, though pg_receivexlog streams WAL
> data in realtime, the latest WAL data might not be available for recovery.

Ah, so you were recovering data from the last, partial, file? Not from
a completed file?

I'm rewriting the handling of partial files per the other thread
started by Heikki. The idea is that there will be an actual .partial
file in there when pg_receivexlog has ended, and you have to deal with
it manually. The typical way would be to pad it with zeroes to the
end. Doing such padding would solve this recovery issue, correct?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/