Re: Updated version of pg_receivexlog

On Wed, Sep 28, 2011 at 08:38, Jaime Casanova <jaime(at)2ndquadrant(dot)com> wrote:
> On Tue, Aug 16, 2011 at 9:32 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> Here's an updated version of pg_receivexlog, that should now actually
>> work (it previously failed miserably when a replication record crossed
>> a WAL file boundary - something which I at the time could not properly
>> reproduce, but when I restarted my work on it now could easily
>> reproduce every time :D).
>>
>> It also contains an update to pg_basebackup that allows it to stream
>> the transaction log in the background while the backup is running,
>> thus reducing the need for wal_keep_segments (if the client can keep
>> up, it should eliminate the need completely).
>>
>
> reviewing this...
>
> i found useful pg_receivexlog as an independent utility, but i'm not
> so sure about the ability to call it from pg_basebackup via --xlog
> option. this is because pg_receivexlog will continue streaming even
> after pg_basebackup if it's called independently but not in the other
> case so the use case for --xlog seems more narrow and error prone (ie:
> you said that it reduces the need for wal_keep_segments *if the client
> can keep up*... how can we know that before starting pg_basebackup?)

These two are not intended to be used together.

pg_basebackup --xlog=stream is intended for the same use-case as
"pg_basebackup -x" today, which is take a backup of just the parts
that you actually need to clone the database, but to do so without
having to guestimate the value for wal_keep_segments.

> pg_receivexlog worked good in my tests.
>
> pg_basebackup with --xlog=stream gives me an already recycled wal
> segment message (note that the file was in pg_xlog in the standby):
> FATAL:  could not receive data from WAL stream: FATAL:  requested WAL
> segment 00000001000000000000005C has already been removed

Do you get this reproducibly? Or did you get it just once?

And when you say "in the standby" what are you referring to? There is
no standby server in the case of pg_basebackup --xlog=stream, it's
just backup... But are you saying pg_basebackup had received the file,
yet tried to get it again?

> in other things:
> do we need to include src/bin/pg_basebackup/.gitignore in the patch?

Not sure what you mean? We need to add pg_receivexlog to this file,
yes - in head it just contains pg_basebackup.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/