Re: Streaming replication and WAL archive interactions

On 03/01/2015 12:36 AM, Venkata Balaji N wrote:
> Patch did get applied successfully to the latest master. Can you please
> rebase.

Here you go.

On 01/31/2015 03:07 PM, Andres Freund wrote:
> On 2014-12-19 22:56:40 +0200, Heikki Linnakangas wrote:
>> This add two new archive_modes, 'shared' and 'always', to indicate whether
>> the WAL archive is shared between the primary and standby, or not. In
>> shared mode, the standby tracks which files have been archived by the
>> primary. The standby refrains from recycling files that the primary has
>> not yet archived, and at failover, the standby archives all those files too
>> from the old timeline. In 'always' mode, the standby's WAL archive is
>> taken to be separate from the primary's, and the standby independently
>> archives all files it receives from the primary.
>
> I don't really like this approach. Sharing a archive is rather dangerous
> in my experience - if your old master comes up again (and writes in the
> last wal file) or similar, you can get into really bad situations.

It doesn't have to actually be shared. The master and standby could
archive to different locations, but the responsibility of archiving is
shared, so that on promotion, the standby ensures that every WAL file
gets archived. If the master didn't do it, then the standby will.

Yes, if the master comes up again, it might try to archive a file that
the standby already archived. But that's not so bad. Both copies of the
file will be identical. You could put logic in archive_command to check,
if the file already exists in the archive, whether the contents are
identical, and return success without doing anything if they are.

Oh, hang on, that's not necessarily true. On promotion, the standby
archives the last, partial WAL segment from the old timeline. That's
just wrong
(http://www.postgresql.org/message-id/52FCD37C.3070806@vmware.com), and
in fact I somehow thought I changed that already, but apparently not. So
let's stop doing that.

> What I was thinking about was instead trying to detect the point up to
> which files were safely archived by running restore command to check for
> the presence of archived files. Then archive anything that has valid
> content and isn't yet archived. That doesn't sound particularly
> complicated to me.

Hmm. That assumes that the standby has a valid restore_command, and can
access the WAL archive. Not a too unreasonable requirement I guess, but
with the scheme I proposed, it's not necessary. Seems a bit silly to
copy a whole segment from the archive just to check if it exists, though.

- Heikki