Re: Updated version of pg_receivexlog

On Thu, Oct 27, 2011 at 10:12, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Thu, Oct 27, 2011 at 4:49 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> On Thu, Oct 27, 2011 at 09:46, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>> On Thu, Oct 27, 2011 at 4:40 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>> On Thu, Oct 27, 2011 at 09:29, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>>>> On Thu, Oct 27, 2011 at 3:29 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>>>> I've applied this version with a few more minor changes that Heikki found.
>>>>>
>>>>> Cool!
>>>>>
>>>>> When I tried pg_receivexlog and checked the contents of streamed WAL file by
>>>>> xlogdump, I found that recent WAL records that walsender has already sent don't
>>>>> exist in that WAL file. I expected that pg_receivexlog writes the streamed WAL
>>>>> records to the disk as soon as possible, but it doesn't. Is this
>>>>> intentional? Or bug?
>>>>> Am I missing something?
>>>>
>>>> It writes it to disk as soon as possible, but doesn't fsync() until
>>>> the end of each segment. Are you by any chance looking at the file
>>>> while it's running?
>>>
>>> No. I looked at that file after shutting down the master server.
>>
>> Ugh, in that case something is certainly wrong. There is nothing but
>> setting up some offset values between PQgetCopyData() and write()...
>
> When end-of-copy stream is found or an error happens, pg_receivexlog
> exits without flushing outstanding WAL records. Which seems to cause
> the problem I reported.

Not sure I follow. When we arrive at PQgetCopyData() there should be
nothing buffered, and if the end of stream happens there it returns
-1, and we exit, no? So where is the data that's lost?

I do realize we don't actually fsync() and close() in this case - is
that what you are referring to? But the data should already have been
write()d, so it should still be there, no?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/