Skip site navigation (1) Skip section navigation (2)

Teaching pg_receivexlog to follow timeline switches

From: Heikki Linnakangas <hlinnakangas(at)vmware(dot)com>
To: PostgreSQL-development <pgsql-hackers(at)postgreSQL(dot)org>
Subject: Teaching pg_receivexlog to follow timeline switches
Date: 2013-01-15 14:05:57
Message-ID: 50F56245.8050802@vmware.com (view raw or flat)
Thread:
Lists: pgsql-hackers
Now that a standby server can follow timeline switches through streaming 
replication, we should do teach pg_receivexlog to do the same. Patch 
attached.

I made one change to the way START_STREAMING command works, to better 
support this. When a standby server reaches the timeline it's streaming 
from the master, it stops streaming, fetches any missing timeline 
history files, and parses the history file of the latest timeline to 
figure out where to continue. However, I don't want to parse timeline 
history files in pg_receivexlog. Better to keep it simple. So instead, I 
modified the server-side code for START_STREAMING to return the next 
timeline's ID at the end, and used that in pg_receivexlog. I also 
modifed BASE_BACKUP to return not only the start XLogRecPtr, but also 
the corresponding timeline ID. Otherwise we might try to start streaming 
from wrong timeline if you issue a BASE_BACKUP at the same moment the 
server switches to a new timeline.

When pg_receivexlog switches timeline, what to do with the partial file 
on the old timeline? When the timeline changes in the middle of a WAL 
segment, the segment old the old timeline is only half-filled. For 
example, when timeline changes from 1 to 2, you'll have this in pg_xlog:

000000010000000000000006
000000010000000000000007
000000010000000000000008
000000020000000000000008
00000002.history

The segment 000000010000000000000008 is only half-filled, as the 
timeline changed in the middle of that segment. The beginning portion of 
that file is duplicated in 000000020000000000000008, with the 
timeline-changing checkpoint record right after the duplicated portion.

When we stream that with pg_receivexlog, and hit the timeline switch, 
we'll have this situation in the client:

000000010000000000000006
000000010000000000000007
000000010000000000000008.partial

What to do with the partial file? One option is to rename it to 
000000010000000000000008. However, if you then kill pg_receivexlog 
before it has finished streaming a full segment from the new timeline, 
on restart it will try to begin streaming WAL segment 
000000010000000000000009, because it sees that segment 
000000010000000000000008 is already completed. That'd be wrong.

The best option seems to be to just leave the .partial file in place, so 
as streaming progresses, you end up with:

000000010000000000000006
000000010000000000000007
000000010000000000000008.partial
000000020000000000000008
000000020000000000000009
00000002000000000000000A.partial

It feels a bit confusing to have that old partial file there, but that 
seems like the most correct solution. That file is indeed partial. This 
also ensures that if the server running on timeline 1 continues to 
generate new WAL, and it fills 000000010000000000000008, we won't 
confuse the partial segment with that name with a full one.

- Heikki

Attachment: teach-receivexlog-to-switch-timelines-1.patch
Description: text/x-diff (50.3 KB)

Responses

pgsql-hackers by date

Next:From: Alvaro HerreraDate: 2013-01-15 14:21:05
Subject: Re: recent ALTER whatever .. SET SCHEMA refactoring
Previous:From: Peter EisentrautDate: 2013-01-15 13:53:30
Subject: Re: another idea for changing global configuration settings from SQL

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group