Re: Your review of pg_receivexlog/pg_basebackup

On Wed, Oct 26, 2011 at 09:52, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> (CC'ing pgsql-hackers, this started as an IM discussion yesterday but really
> belongs in the archives)
>
> On 25.10.2011 23:52, Magnus Hagander wrote:
>>>
>>> There's a tiny chance to get incomplete xlog files with pg_receivexlog if
>>> you crash:
>>> 1. pg_receivexlog finishes write()ing a file but system crashes before
>>> fsync() finishes.
>>> 2. When pg_receivexlog restarts after crash, the last WAL file was not
>>> fully flushed to disk, with
>>> holes in the middle, but it has the right length. pg_receivexlog will
>>> continue streaming from the next file.
>>> not sure if we care about such a narrow window, but maybe we do
>>
>> So how would we go about fixing that?  Always unlink the last file in
>> the directory and try from there would seem dangerous too - what if
>> it's not available on the master anymore, then we might have given up
>> on data...
>
> Start streaming from the beginning of the last segment, but don't unlink it
> first. Just overwrite it as you receive the data.
>
> Or, always create new xlog file as "0000000100000001000000D3.partial", and
> only when it's fully written, fsync it, and then rename it to
> "0000000100000001000000D3". Then you know that if a file doesn't have the
> .partial suffix, it's complete. The fact that the last partial file always
> has the .partial suffix needs some extra pushups in the restore_command,
> though.

Here's a version that does this. Turns out this requires a lot less
code than what was previously in there, which is always nice.

We still need to solve the other part which is how to deal with the
partial files on restore. But this is definitely a cleaner way from a
pure pg_receivexlog perspective.

Comments/reviews?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/