Skip site navigation (1) Skip section navigation (2)

Re: pg_basebackup from cascading standby after timeline switch

From: Heikki Linnakangas <hlinnakangas(at)vmware(dot)com>
To: Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
Cc: Magnus Hagander <magnus(at)hagander(dot)net>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: pg_basebackup from cascading standby after timeline switch
Date: 2013-01-02 13:55:35
Message-ID: 50E43C57.5050101@vmware.com (view raw or flat)
Thread:
Lists: pgsql-hackers
On 27.12.2012 12:06, Heikki Linnakangas wrote:
> On 23.12.2012 15:33, Fujii Masao wrote:
>> On Fri, Dec 21, 2012 at 9:54 PM, Heikki Linnakangas
>> <hlinnakangas(at)vmware(dot)com> wrote:
>>> Yes, this should be backpatched to 9.2. I came up with the attached.
>>
>> In this patch, if '-X stream' is specified in pg_basebackup, the timeline
>> history files are not backed up.
>
> Good point.
>
>> We should change pg_backup background
>> process and walsender so that they stream also timeline history files,
>> for example, by using 'TIMELINE_HISTORY' replication command?
>> Or basebackup.c should send all timeline history files at the end of
>> backup
>> even if '-X stream' is specified?
>
> Perhaps. We should enhance pg_receivexlog to follow timeline switches,
> anyway. I was thinking of leaving that as a todo item, but pg_basebackup
> -X stream shares the code, so we should implement that now to get that
> support into both.
>
> In the problem you reported on the other thread
> (http://archives.postgresql.org/message-id/50DB5EA9.7010406@vmware.com),
> you also need the timeline history files, but that one didn't use "-X"
> at all. Even if we teach pg_basebackup to fetch the timeline history
> files in "-X stream" mode, that still leaves the problem on that other
> thread.
>
> The simplest solution would be to always include all timeline history
> files in the backup, even if -X is not used. Currently, however, pg_xlog
> is backed up as an empty directory in that case, but that would no
> longer be the case if we start including timeline history files there. I
> wonder if that would confuse any existing backup scripts people are using.

This thread has spread out a bit, so here's a summary of the remaining 
issues and what I'm going to do about them:

9.2
---

If you take a backup with "pg_basebackup -X fetch", and the timeline 
switches while the backup is taken, you currently get an error like 
"requested WAL segment 00000001000000000000000C has already been 
removed". To fix, let's change the server-side support of "-X fetch" to 
include all WAL files between the backup start and end pointers, 
regardless of timelines. I'm thinking of doing this by scanning pg_xlog 
with readdir(), and sending over any files in that range. Another option 
would be to read and parse the timeline history file to figure out the 
exact filenames expected, but the readdir() approach seems simpler.

You also need the timeline history files. With "-X fetch", I think we 
should always include them in the pg_xlog directory of the backup, along 
with the WAL files themselves.

"-X stream" has a similar problem. If timeline changes during backup, 
the replication will stop at the timeline switch, and the backup fails. 
There isn't much we can do about that, as you can't follow a timeline 
switch via streaming replication in 9.2. At best, we could try to detect 
the situation and give a better error message.

With plain pg_basebackup, without -X option, you usually need a WAL 
archive to restore. The only exception is when you initialize a 
streaming standby with plain "pg_basebackup", intending to connect it to 
the master right after taking the backup, so that it can stream all the 
required WAL at that point. We have a problem with that scenario, 
because even if there was no timeline switch while the backup was taken 
(if there was, you're screwed the same as with "-X stream"), because of 
the issue mentioned in the first post in this thread: the beginning of 
the first WAL file contains the old timeline ID. Even though that's not 
replayed, because the checkpoint is in the later part of the segment, 
recovery still complains if there is a timeline ID in the beginning of 
the file that we don't recognize as our ancestor. This could be fixed if 
we somehow got the timeline history files in the backup, but I'm worried 
that might break people's backup scripts. At the moment, the pg_xlog 
directory in the backup is empty when -X is not used, we even spell that 
out explicitly in the manual. Including timeline history files would 
change that. That might be an issue if you symlink pg_xlog to a 
different drive, for example. To make things worse, there is no timeline 
history file for timeline 1, so you would not notice when you test your 
backup scripts in simple cases.

In summary, in 9.2 I think we should fix "-X fetch" to tolerate a 
timeline switch, and include all the timeline history files. The 
behavior of other modes would not be changed, and they will fail if a 
timeline changes during or just before backup.

Master
------

In master, we can try harder for the "-X stream" case, because you can 
replicate a timeline switch, and fetch timeline history files via a 
replication connection. Let's teach pg_receivexlog, and "pg_basebackup 
-X stream", to use those facilities, so that even if the timeline 
changes while the backup is taken, all the history files and WAL files 
are correctly included in the backup. I'll start working on a patch to 
do that.

That leaves one case not covered: If you take a backup with plain 
"pg_basebackup" from a standby, without -X, and the first WAL segment 
contains a timeline switch (ie. you take the backup right after a 
failover), and you try to recover from it without a WAL archive, it 
doesn't work. This is the original issue that started this thread, 
except that I used "-x" in my original test case. The problem here is 
that even though streaming replication will fetch the timeline history 
file when it connects, at the very beginning of recovery, we expect that 
we already have the timeline history file corresponding the initial 
timeline available, either in pg_xlog or the WAL archive. By the time 
streaming replication has connected and fetched the history file, we've 
already initialized expectedTLEs to contain just the latest TLI. To fix 
that, we should delay calling readTimeLineHistoryFile() until streaming 
replication has connected and fetched the file.

Barring objections, I'll commit the attached two patches. 
include-wal-files-from-all-timelines-in-base-backup-1.patch is for 9.2 
and master, and it fixes the "pg_basebackup -X fetch" case. 
delay-reading-timeline-history-file.patch is for master, and it changes 
recovery so if a timeline history file for the initial target timeline 
is fetched over streaming replication, expectedTLEs is initialized with 
the streamed file. That fixes the plain "pg_basebackup" without -X case 
on master.

What remains is to teach "pg_receivexlog" and "pg_basebackup -X stream" 
to cross timeline changes. I'll start working on a patch for that.

- Heikki

Attachment: include-wal-files-from-all-timelines-in-base-backup-1.patch
Description: text/x-diff (12.0 KB)
Attachment: delay-reading-timeline-history-file.patch
Description: text/x-diff (8.4 KB)

In response to

Responses

pgsql-hackers by date

Next:From: Noah MischDate: 2013-01-02 14:15:03
Subject: Re: default SSL compression (was: libpq compression)
Previous:From: Magnus HaganderDate: 2013-01-02 13:25:48
Subject: Re: pgsql: Unify some tar functionality across different parts

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group