Fwd: repmgr won't update witness after failover

---------- Forwarded message ----------
From: Aviel Buskila <aviel33(at)gmail(dot)com>
Date: 2015-08-13 15:43 GMT+03:00
Subject: Re: [GENERAL] repmgr won't update witness after failover
To: Jony Cohen ‫‎<jony(dot)cohenjo(at)gmail(dot)com>‎‬

Hey,
I have just tried to start the repmgrd on the new standby after I have
fixed it as a standby and still this goes the same way.

from the message given in the repmgrd log in the witness server it seems
that he is not able to elect a new master because he can't see anyone .

I have check in the repl_nodes table in the witness and it shows:
witness node3
master node2
master node1

is there a way update the witness after the first failover?

2015-08-13 15:06 GMT+03:00 Jony Cohen <jony(dot)cohenjo(at)gmail(dot)com>:

> Hi Aviel,
> you can use the 'show cluster' command to see the repmgr state before you
> do the 2nd failover - make sure the node1 is indeed marked as replica.
> After a failover the Master doesn't automatically attach to the new master
> - you need to point him as a slave (standby follow - if possible...)
> did you start the repmgrd on node1 after making it a replica of the new
> master? (it needs 2 daemons to decide what to promote)
>
> Regards,
> - Jony
>

On Thu, Aug 13, 2015 at 1:29 PM, Aviel Buskila <aviel33(at)gmail(dot)com> wrote:

> Hey,
> I have set up three nodes of postgresql 9.4 with repmgr in this way:
> 1. master - node1
> 2. standby - node2
> 3. witness - node3
>
> Now I have set up the replication and the witness as it says here:
> https://github.com/2ndQuadrant/repmgr/blob/master/FAILOVER.rst
>
> Now when I do 'kill -9 $(pidof postmaster)' The witness detects that
> something went wrong and fails over from node1 to node2
> But when I setup the replication now to work from node2 to node1 and I
> kill the postgresql process it doesn't failover and the repmgrd log shows
> the following message:
> unable to determine a valid master server; waiting 10 seconds to retry...
>
> it seems that the witness doesn't know about the new standby server..
>
> Has anyone got any idea about what am I doing wrong here?
>
>
> Best regards,
> Aviel Buskila
>