Skip site navigation (1)
Skip section navigation (2)
Peripheral Links
Re: Teaching pg_receivexlog to follow timeline switches
Now that a standby server can follow timeline switches through streaming replication, we should do teach pg_receivexlog to do the same. Patch attached. I made one change to the way START_STREAMING command works, to better support this. When a standby server reaches the timeline it's streaming from the master, it stops streaming, fetches any missing timeline history files, and parses the history file of the latest timeline to figure out where to continue. However, I don't want to parse timeline history files in pg_receivexlog. Better to keep it simple. So instead, I modified the server-side code for START_STREAMING to return the next timeline's ID at the end, and used that in pg_receivexlog. I also modifed BASE_BACKUP to return not only the start XLogRecPtr, but also the corresponding timeline ID. Otherwise we might try to start streaming from wrong timeline if you issue a BASE_BACKUP at the same moment the server switches to a new timeline. When pg_receivexlog switches timeline, what to do with the partial file on the old timeline? When the timeline changes in the middle of a WAL segment, the segment old the old timeline is only half-filled. For example, when timeline changes from 1 to 2, you'll have this in pg_xlog: 000000010000000000000006 000000010000000000000007 000000010000000000000008 000000020000000000000008 00000002.history The segment 000000010000000000000008 is only half-filled, as the timeline changed in the middle of that segment. The beginning portion of that file is duplicated in 000000020000000000000008, with the timeline-changing checkpoint record right after the duplicated portion. When we stream that with pg_receivexlog, and hit the timeline switch, we'll have this situation in the client: 000000010000000000000006 000000010000000000000007 000000010000000000000008.partial What to do with the partial file? One option is to rename it to 000000010000000000000008. However, if you then kill pg_receivexlog before it has finished streaming a full segment from the new timeline, on restart it will try to begin streaming WAL segment 000000010000000000000009, because it sees that segment 000000010000000000000008 is already completed. That'd be wrong. The best option seems to be to just leave the .partial file in place, so as streaming progresses, you end up with: 000000010000000000000006 000000010000000000000007 000000010000000000000008.partial 000000020000000000000008 000000020000000000000009 00000002000000000000000A.partial It feels a bit confusing to have that old partial file there, but that seems like the most correct solution. That file is indeed partial. This also ensures that if the server running on timeline 1 continues to generate new WAL, and it fills 000000010000000000000008, we won't confuse the partial segment with that name with a full one. - Heikki
Attachment: teach-receivexlog-to-switch-timelines-1.patch
Description: text/x-diff (50.3 KB)
Description: text/x-diff (50.3 KB)
On Tue, Jan 15, 2013 at 11:05 PM, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> wrote: > Now that a standby server can follow timeline switches through streaming > replication, we should do teach pg_receivexlog to do the same. Patch > attached. > > I made one change to the way START_STREAMING command works, to better > support this. When a standby server reaches the timeline it's streaming from > the master, it stops streaming, fetches any missing timeline history files, > and parses the history file of the latest timeline to figure out where to > continue. However, I don't want to parse timeline history files in > pg_receivexlog. Better to keep it simple. So instead, I modified the > server-side code for START_STREAMING to return the next timeline's ID at the > end, and used that in pg_receivexlog. I also modifed BASE_BACKUP to return > not only the start XLogRecPtr, but also the corresponding timeline ID. > Otherwise we might try to start streaming from wrong timeline if you issue a > BASE_BACKUP at the same moment the server switches to a new timeline. > > When pg_receivexlog switches timeline, what to do with the partial file on > the old timeline? When the timeline changes in the middle of a WAL segment, > the segment old the old timeline is only half-filled. For example, when > timeline changes from 1 to 2, you'll have this in pg_xlog: > > 000000010000000000000006 > 000000010000000000000007 > 000000010000000000000008 > 000000020000000000000008 > 00000002.history > > The segment 000000010000000000000008 is only half-filled, as the timeline > changed in the middle of that segment. The beginning portion of that file is > duplicated in 000000020000000000000008, with the timeline-changing > checkpoint record right after the duplicated portion. > > When we stream that with pg_receivexlog, and hit the timeline switch, we'll > have this situation in the client: > > 000000010000000000000006 > 000000010000000000000007 > 000000010000000000000008.partial > > What to do with the partial file? One option is to rename it to > 000000010000000000000008. However, if you then kill pg_receivexlog before it > has finished streaming a full segment from the new timeline, on restart it > will try to begin streaming WAL segment 000000010000000000000009, because it > sees that segment 000000010000000000000008 is already completed. That'd be > wrong. Can't we rename .partial file safely after we receive a full segment of the WAL file with new timeline and the same logid/segmentid? Regards, -- Fujii Masao
On 15.01.2013 20:22, Fujii Masao wrote: > On Tue, Jan 15, 2013 at 11:05 PM, Heikki Linnakangas > <hlinnakangas(at)vmware(dot)com> wrote: >> Now that a standby server can follow timeline switches through streaming >> replication, we should do teach pg_receivexlog to do the same. Patch >> attached. >> >> I made one change to the way START_STREAMING command works, to better >> support this. When a standby server reaches the timeline it's streaming from >> the master, it stops streaming, fetches any missing timeline history files, >> and parses the history file of the latest timeline to figure out where to >> continue. However, I don't want to parse timeline history files in >> pg_receivexlog. Better to keep it simple. So instead, I modified the >> server-side code for START_STREAMING to return the next timeline's ID at the >> end, and used that in pg_receivexlog. I also modifed BASE_BACKUP to return >> not only the start XLogRecPtr, but also the corresponding timeline ID. >> Otherwise we might try to start streaming from wrong timeline if you issue a >> BASE_BACKUP at the same moment the server switches to a new timeline. >> >> When pg_receivexlog switches timeline, what to do with the partial file on >> the old timeline? When the timeline changes in the middle of a WAL segment, >> the segment old the old timeline is only half-filled. For example, when >> timeline changes from 1 to 2, you'll have this in pg_xlog: >> >> 000000010000000000000006 >> 000000010000000000000007 >> 000000010000000000000008 >> 000000020000000000000008 >> 00000002.history >> >> The segment 000000010000000000000008 is only half-filled, as the timeline >> changed in the middle of that segment. The beginning portion of that file is >> duplicated in 000000020000000000000008, with the timeline-changing >> checkpoint record right after the duplicated portion. >> >> When we stream that with pg_receivexlog, and hit the timeline switch, we'll >> have this situation in the client: >> >> 000000010000000000000006 >> 000000010000000000000007 >> 000000010000000000000008.partial >> >> What to do with the partial file? One option is to rename it to >> 000000010000000000000008. However, if you then kill pg_receivexlog before it >> has finished streaming a full segment from the new timeline, on restart it >> will try to begin streaming WAL segment 000000010000000000000009, because it >> sees that segment 000000010000000000000008 is already completed. That'd be >> wrong. > > Can't we rename .partial file safely after we receive a full segment > of the WAL file > with new timeline and the same logid/segmentid? I'd prefer to leave the .partial suffix in place, as the segment really isn't complete. It doesn't make a difference when you recover to the latest timeline, but if you have a more complicated scenario with multiple timelines that are still "alive", ie. there's a server still actively generating WAL on that timeline, you'll easily get confused. As an example, imagine that you have a master server, and one standby. You maintain a WAL archive for backup purposes with pg_receivexlog, connected to the standby. Now, for some reason, you get a split-brain situation and the standby server is promoted with new timeline 2, while the real master is still running. The DBA notices the problem, and kills the standby and pg_receivexlog. He deletes the XLOG files belonging to timeline 2 in pg_receivexlog's target directory, and re-points pg_recevexlog to the master while he re-builds the standby server from backup. At that point, pg_receivexlog will start streaming from the end of the zero-padded segment, not knowing that it was partial, and you have a hole in the archived WAL stream. Oops. The DBA could avoid that by also removing the last WAL segment on timeline 1, the one that was partial. But it's really not obvious that there's anything wrong with that segment. Keeping the .partial suffix makes it clear. - Heikki
On Thu, Jan 17, 2013 at 1:08 AM, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> wrote: > On 15.01.2013 20:22, Fujii Masao wrote: >> >> On Tue, Jan 15, 2013 at 11:05 PM, Heikki Linnakangas >> <hlinnakangas(at)vmware(dot)com> wrote: >>> >>> Now that a standby server can follow timeline switches through streaming >>> replication, we should do teach pg_receivexlog to do the same. Patch >>> attached. >>> >>> I made one change to the way START_STREAMING command works, to better >>> support this. When a standby server reaches the timeline it's streaming >>> from >>> the master, it stops streaming, fetches any missing timeline history >>> files, >>> and parses the history file of the latest timeline to figure out where to >>> continue. However, I don't want to parse timeline history files in >>> pg_receivexlog. Better to keep it simple. So instead, I modified the >>> server-side code for START_STREAMING to return the next timeline's ID at >>> the >>> end, and used that in pg_receivexlog. I also modifed BASE_BACKUP to >>> return >>> not only the start XLogRecPtr, but also the corresponding timeline ID. >>> Otherwise we might try to start streaming from wrong timeline if you >>> issue a >>> BASE_BACKUP at the same moment the server switches to a new timeline. >>> >>> When pg_receivexlog switches timeline, what to do with the partial file >>> on >>> the old timeline? When the timeline changes in the middle of a WAL >>> segment, >>> the segment old the old timeline is only half-filled. For example, when >>> timeline changes from 1 to 2, you'll have this in pg_xlog: >>> >>> 000000010000000000000006 >>> 000000010000000000000007 >>> 000000010000000000000008 >>> 000000020000000000000008 >>> 00000002.history >>> >>> The segment 000000010000000000000008 is only half-filled, as the timeline >>> changed in the middle of that segment. The beginning portion of that file >>> is >>> duplicated in 000000020000000000000008, with the timeline-changing >>> checkpoint record right after the duplicated portion. >>> >>> When we stream that with pg_receivexlog, and hit the timeline switch, >>> we'll >>> have this situation in the client: >>> >>> 000000010000000000000006 >>> 000000010000000000000007 >>> 000000010000000000000008.partial >>> >>> What to do with the partial file? One option is to rename it to >>> 000000010000000000000008. However, if you then kill pg_receivexlog before >>> it >>> has finished streaming a full segment from the new timeline, on restart >>> it >>> will try to begin streaming WAL segment 000000010000000000000009, because >>> it >>> sees that segment 000000010000000000000008 is already completed. That'd >>> be >>> wrong. >> >> >> Can't we rename .partial file safely after we receive a full segment >> of the WAL file >> with new timeline and the same logid/segmentid? > > > I'd prefer to leave the .partial suffix in place, as the segment really > isn't complete. It doesn't make a difference when you recover to the latest > timeline, but if you have a more complicated scenario with multiple > timelines that are still "alive", ie. there's a server still actively > generating WAL on that timeline, you'll easily get confused. > > As an example, imagine that you have a master server, and one standby. You > maintain a WAL archive for backup purposes with pg_receivexlog, connected to > the standby. Now, for some reason, you get a split-brain situation and the > standby server is promoted with new timeline 2, while the real master is > still running. The DBA notices the problem, and kills the standby and > pg_receivexlog. He deletes the XLOG files belonging to timeline 2 in > pg_receivexlog's target directory, and re-points pg_recevexlog to the master > while he re-builds the standby server from backup. At that point, > pg_receivexlog will start streaming from the end of the zero-padded segment, > not knowing that it was partial, and you have a hole in the archived WAL > stream. Oops. > > The DBA could avoid that by also removing the last WAL segment on timeline > 1, the one that was partial. But it's really not obvious that there's > anything wrong with that segment. Keeping the .partial suffix makes it > clear. Thanks for elaborating the reason why .partial suffix should be kept. I agree that keeping the .partial suffix would be safer. Regards, -- Fujii Masao
Fujii Masao <masao(dot)fujii(at)gmail(dot)com> writes: > Thanks for elaborating the reason why .partial suffix should be kept. > I agree that keeping the .partial suffix would be safer. +1 to both points. So +2 I guess :) Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On Wed, Jan 16, 2013 at 11:08 AM, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> wrote: > I'd prefer to leave the .partial suffix in place, as the segment really > isn't complete. It doesn't make a difference when you recover to the latest > timeline, but if you have a more complicated scenario with multiple > timelines that are still "alive", ie. there's a server still actively > generating WAL on that timeline, you'll easily get confused. > > As an example, imagine that you have a master server, and one standby. You > maintain a WAL archive for backup purposes with pg_receivexlog, connected to > the standby. Now, for some reason, you get a split-brain situation and the > standby server is promoted with new timeline 2, while the real master is > still running. The DBA notices the problem, and kills the standby and > pg_receivexlog. He deletes the XLOG files belonging to timeline 2 in > pg_receivexlog's target directory, and re-points pg_recevexlog to the master > while he re-builds the standby server from backup. At that point, > pg_receivexlog will start streaming from the end of the zero-padded segment, > not knowing that it was partial, and you have a hole in the archived WAL > stream. Oops. > > The DBA could avoid that by also removing the last WAL segment on timeline > 1, the one that was partial. But it's really not obvious that there's > anything wrong with that segment. Keeping the .partial suffix makes it > clear. I shudder at the idea that the DBA is manually involved in any of this. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 17.01.2013 16:56, Robert Haas wrote: > On Wed, Jan 16, 2013 at 11:08 AM, Heikki Linnakangas > <hlinnakangas(at)vmware(dot)com> wrote: >> I'd prefer to leave the .partial suffix in place, as the segment really >> isn't complete. It doesn't make a difference when you recover to the latest >> timeline, but if you have a more complicated scenario with multiple >> timelines that are still "alive", ie. there's a server still actively >> generating WAL on that timeline, you'll easily get confused. >> >> As an example, imagine that you have a master server, and one standby. You >> maintain a WAL archive for backup purposes with pg_receivexlog, connected to >> the standby. Now, for some reason, you get a split-brain situation and the >> standby server is promoted with new timeline 2, while the real master is >> still running. The DBA notices the problem, and kills the standby and >> pg_receivexlog. He deletes the XLOG files belonging to timeline 2 in >> pg_receivexlog's target directory, and re-points pg_recevexlog to the master >> while he re-builds the standby server from backup. At that point, >> pg_receivexlog will start streaming from the end of the zero-padded segment, >> not knowing that it was partial, and you have a hole in the archived WAL >> stream. Oops. >> >> The DBA could avoid that by also removing the last WAL segment on timeline >> 1, the one that was partial. But it's really not obvious that there's >> anything wrong with that segment. Keeping the .partial suffix makes it >> clear. > > I shudder at the idea that the DBA is manually involved in any of this. The scenario I described is that you screwed up your failover environment, and end up with a split-brain situation by accident. The DBA certainly needs to be involved to recover from that. - Heikki
On Thu, Jan 17, 2013 at 9:59 AM, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> wrote: > The scenario I described is that you screwed up your failover environment, > and end up with a split-brain situation by accident. The DBA certainly needs > to be involved to recover from that. OK, I agree, but I still think a lot of DBAs would have no idea how to handle that situation. I agree with your proposal, don't get me wrong - I just think there's still an awful lot of room for operator error in these more complex replication scenarios. I don't have a clue how to fix that, and it's certainly not the purpose of this thread to fix that; I'm just venting. Actually, I'm really glad to see all the work you've done to improve the way that some of these scenarios work and eliminate various bugs and other surprising failure modes over the last couple of months. It's great stuff. Alas, I think we still some distance from being able to provide an "easy button". -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas escribió: > Actually, I'm really glad to see all the work you've done to improve > the way that some of these scenarios work and eliminate various bugs > and other surprising failure modes over the last couple of months. > It's great stuff. +1 -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Tue, Jan 15, 2013 at 9:05 AM, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> wrote: > Now that a standby server can follow timeline switches through streaming > replication, we should do teach pg_receivexlog to do the same. Patch > attached. Is it possible to re-use walreceiver code from the backend? I was thinking that it would actually be very useful to have the whole replication functionality modularized and in a standalone binary that could act as a replication proxy and WAL archiver that could run without all the overhead of an entire PG instance.
On 18.01.2013 06:38, Phil Sorber wrote: > On Tue, Jan 15, 2013 at 9:05 AM, Heikki Linnakangas > <hlinnakangas(at)vmware(dot)com> wrote: >> Now that a standby server can follow timeline switches through streaming >> replication, we should do teach pg_receivexlog to do the same. Patch >> attached. > > Is it possible to re-use walreceiver code from the backend? > > I was thinking that it would actually be very useful to have the whole > replication functionality modularized and in a standalone binary that > could act as a replication proxy and WAL archiver that could run > without all the overhead of an entire PG instance There's much sense in trying to extract that into a stand-along module. src/bin/pg_basebackup/receivelog.c is about 1000 lines of code at the moment, and it looks quite different from the corresponding code in the backend, because it doesn't have all the backend infrastructure available. - Heikki
On Fri, Jan 18, 2013 at 7:55 AM, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> wrote: > On 18.01.2013 06:38, Phil Sorber wrote: >> Is it possible to re-use walreceiver code from the backend? >> >> I was thinking that it would actually be very useful to have the whole >> replication functionality modularized and in a standalone binary that >> could act as a replication proxy and WAL archiver that could run >> without all the overhead of an entire PG instance > > > There's much sense in trying to extract that into a stand-along module. > src/bin/pg_basebackup/receivelog.c is about 1000 lines of code at the > moment, and it looks quite different from the corresponding code in the > backend, because it doesn't have all the backend infrastructure available. > > - Heikki That's fair. What do you think about the idea of a full WAL proxy? Probably not for 9.3 at this point though.
This patch was in Needs Review status, but you committed it on 2013-01-17. I have marked it as such in the CF app.
Phil Sorber <phil(at)omniti(dot)com> writes: > What do you think about the idea of a full WAL proxy? Probably not for > 9.3 at this point though. I was thinking that a WAL proxy nowadays is called a cascading standby with local archiving enabled. I'm not sure why you would want to trust your archiving and WAL relaying to another piece of software… Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On 22.01.2013 15:02, Dimitri Fontaine wrote: > Phil Sorber<phil(at)omniti(dot)com> writes: >> What do you think about the idea of a full WAL proxy? Probably not for >> 9.3 at this point though. > > I was thinking that a WAL proxy nowadays is called a cascading standby > with local archiving enabled. I'm not sure why you would want to trust > your archiving and WAL relaying to another piece of software… You might not want to keep a copy of the whole data directory around, as you have to in a cascading standby. I can see value in a separate WAL proxy software, especially if it's integrated into a larger backup manager program like barman or wal-e. - Heikki
Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> writes: > You might not want to keep a copy of the whole data directory around, as you > have to in a cascading standby. I can see value in a separate WAL proxy > software, especially if it's integrated into a larger backup manager program > like barman or wal-e. +1 I somehow forgot about $PGDATA here. Time for a little break I guess :) Another idea is to have a daemon mode pg_receivexlog where not only it can maintain a local archive but also feed it using the replication protocol to standbies, keeping track of their position. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On Tue, Jan 22, 2013 at 8:33 AM, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr> wrote: > Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> writes: >> You might not want to keep a copy of the whole data directory around, as you >> have to in a cascading standby. I can see value in a separate WAL proxy >> software, especially if it's integrated into a larger backup manager program >> like barman or wal-e. > > +1 > > I somehow forgot about $PGDATA here. Time for a little break I guess :) > > Another idea is to have a daemon mode pg_receivexlog where not only it > can maintain a local archive but also feed it using the replication > protocol to standbies, keeping track of their position. I'm not sure if i described it well, but that's essentially what I was asking about. It would have both wal receiving and and wal sending capability. Along with it's own local WAL storage perhaps governed in size by a keep_wal_segments and also a longer term archive that you could have compressed but also pull from with a archive and restore command. And also be able to act as a synchronous replication peer. I think it has already been discussed to have pg_receivexlog do that last one. So yeah, a cascading standby without $PGDATA or hot_standby or large shared_buffers resources. It seems like maybe we could add through subtraction. Add a parameter that disables wal replay? I'm sure there'd be more things it would have to disable, but then it's not two separate binaries. > > Regards, > -- > Dimitri Fontaine > http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On 01/22/2013 06:43 AM, Noah Misch wrote: > This patch was in Needs Review status, but you committed it on 2013-01-17. I > have marked it as such in the CF app. Thankyou. There's a lot to keep up with :S -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services