Re: Reliable WAL file shipping over unreliable network

From: Stephen Frost <sfrost(at)snowman(dot)net>
To: Rui DeSousa <rui(dot)desousa(at)icloud(dot)com>
Cc: scott ribe <scott_ribe(at)elevated-dev(dot)com>, Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz>, Dianne Skoll <dfs(at)roaringpenguin(dot)com>, pgsql-admin(at)lists(dot)postgresql(dot)org
Subject: Re: Reliable WAL file shipping over unreliable network
Date: 2018-03-02 18:09:11
Message-ID: 20180302180911.GO2416@tamriel.snowman.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

Greetings,

* Rui DeSousa (rui(dot)desousa(at)icloud(dot)com) wrote:
> > On Mar 1, 2018, at 12:21 AM, scott ribe <scott_ribe(at)elevated-dev(dot)com> wrote:
> > The false report of success is not good, but it's not the root problem.
>
> A false success if a problem; especially in this use case as the source WAL file will be deleted by Postgres before it was truly successful. While monitoring is nice to avoid the issue it is not a fix for the issue.
>
> I personally cannot recommend the use of rsync in this application for two reasons.

There's a bigger problem that I'm amazed hasn't been brought up already-
the rsync, copy, cp, whatever, may finish just fine and then the system
that the WAL file was copied to crashes and you end up losing a bunch of
WAL because none of these methods actually sync's the WAL to disk.

No archive_command should ever return success until the WAL is actually
written out to permanent storage and sync'd. That's what tools like
pgBackRest do and is part of the reason why people really shouldn't try
to hack up their own archive_command solution but should be using well
tested existing solutions.

Thanks!

Stephen

In response to

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Rui DeSousa 2018-03-02 18:22:53 Re: Reliable WAL file shipping over unreliable network
Previous Message Tom Lane 2018-03-02 17:14:49 Re: What is the accepted practice to automate initdb (PostgreSQL 9.6) to a non-default directory?