Re: Your review of pg_receivexlog/pg_basebackup

(CC'ing pgsql-hackers, this started as an IM discussion yesterday but
really belongs in the archives)

On 25.10.2011 23:52, Magnus Hagander wrote:
>> There's a tiny chance to get incomplete xlog files with pg_receivexlog if you crash:
>> 1. pg_receivexlog finishes write()ing a file but system crashes before fsync() finishes.
>> 2. When pg_receivexlog restarts after crash, the last WAL file was not fully flushed to disk, with
>> holes in the middle, but it has the right length. pg_receivexlog will continue streaming from the next file.
>> not sure if we care about such a narrow window, but maybe we do
>
> So how would we go about fixing that? Always unlink the last file in
> the directory and try from there would seem dangerous too - what if
> it's not available on the master anymore, then we might have given up
> on data...

Start streaming from the beginning of the last segment, but don't unlink
it first. Just overwrite it as you receive the data.

Or, always create new xlog file as "0000000100000001000000D3.partial",
and only when it's fully written, fsync it, and then rename it to
"0000000100000001000000D3". Then you know that if a file doesn't have
the .partial suffix, it's complete. The fact that the last partial file
always has the .partial suffix needs some extra pushups in the
restore_command, though.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com