Re: WAL "low watermark" during base backup

On Sun, Sep 4, 2011 at 19:02, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Fri, Sep 2, 2011 at 6:52 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>
>> Attached patch implements a "low watermark wal location" in the
>> walsender shmem array. Setting this value in a walsender prevents
>> transaction log removal prior to this point - similar to how
>> wal_keep_segments work, except with an absolute number rather than
>> relative. For now, this is set when running a base backup with WAL
>> included - to prevent the required WAL to be recycled away while the
>> backup is running, without having to guestimate the value for
>> wal_keep_segments. (There could be other ways added to set it in the
>> future, but that's the only one I've done for now)
>>
>> It obviously needs some documentation updates as well, but I wanted to
>> get some comments on the way it's done before I work on those.
>
> I'm not yet fully available for a discussion on this, but not sure I like this.
>
> You don't have to guess the setting of wal_keep_segments, you
> calculate it exactly from the size of your WAL disk. No other
> calculation is easy or accurate.

Uh, no. What about the (very large number of) cases where pg is just
sitting on one partition, possibly shared with a whole lot of other
services? You'd need to set it to all-of-your-disk, which is something
that will change over time.

Maybe I wasn't entirely clear in the submission, but if it wasn't
obvious: the use-case for this is the small and simple installations
that need a simple way of doing a reliable online backup. This is the
"pg_basebackup -x" usecase altogether - for example, anybody "bigger"
likely has archiv elogging setup already, in which case this
functionality is not interesting at all.

> This patch implements "fill disk until primary croaks" behaviour which
> means you are making a wild and risky guess as to whether it will
> work. If it does not, you are hosed.

Replace "primary" with "server" - remember that this is about backups
and not replication primarily.

That said, you are correct, it does implement that. But then again,
logging into the database and opening a transaction and just leaving
it around for $forever will have similar problems - yet, we allow
users to do that.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/