Using old master as new replica after clean switchover

From: Nikolay Samokhvalov <samokhvalov(at)gmail(dot)com>
To: pgsql-docs(at)lists(dot)postgresql(dot)org, pgsql-hackers(at)postgresql(dot)org
Subject: Using old master as new replica after clean switchover
Date: 2018-10-25 06:57:18
Message-ID: CANNMO+KYuH3Gh7BZp=UGXpoos4tBR0AFgoONkqWBrokuJthEug@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-docs pgsql-hackers

Currently, the documentation explicitly states, that after failover, the
old master must be recreated from scratch, or pg_rewind should be used
(requiring wal_log_hints to be on, which is off by default):

> The former standby is now the primary, but the former primary is down and
might stay down. To return to normal operation, a standby server must be
recreated, either on the former primary system when it comes up, or on a
third, possibly new, system. The pg_rewind utility can be used to speed up
this process on large clusters.

My research shows that some people already rely on the following when
planned failover (aka switchover) procedure, doing it in production:

1) shutdown the current master
2) ensure that the "master candidate" replica has received all WAL data
including shutdown checkpoint from the old master
3) promote the master candidate to make it new master
4) configure recovery.conf on the old master node, while it's inactive
5) start the old master node as a new replica following the new master.

It looks to me now, that if no steps missed in the procedure, this approach
is eligible for Postgres versions 9.3+ (for older versions like 9.3 maybe
not really always – people who know details better will correct me here
maybe). Am I right? Or I'm missing some risks here?

Two changes were made in 9.3 which allowed this approach in general [1]
[2]. Also, I see from the code [3] that during shutdown process, the
walsenders are the last who are stopped, so allow replicas to get the
shutdown checkpoint information.

Is this approach considered as safe now?

if so, let's add it to the documentation, making it official. The patch is
attached.

Links:
[0] 26.3 Failover
https://www.postgresql.org/docs/current/static/warm-standby-failover.html
[1] Support clean switchover
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=985bd7d49726c9f178558491d31a570d47340459
[2] Allow a streaming replication standby to follow a timeline switch
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=abfd192b1b5ba5216ac4b1f31dcd553106304b19
[3]
https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/replication/walsender.c;hb=HEAD#l276

Regards,
Nik

Attachment Content-Type Size
failover_doc.patch application/octet-stream 1.1 KB

Responses

Browse pgsql-docs by date

  From Date Subject
Next Message Jehan-Guillaume de Rorthais 2018-10-25 09:15:51 Re: Using old master as new replica after clean switchover
Previous Message PG Doc comments form 2018-10-24 12:27:24 COPY on partitioned table

Browse pgsql-hackers by date

  From Date Subject
Next Message MyungKyu LIM 2018-10-25 07:28:13 RE: [Todo item] Add entry creation timestamp column to pg_stat_replication
Previous Message Fabien COELHO 2018-10-25 06:21:27 Re: pgbench - add pseudo-random permutation function