Re: Slave enters in recovery and promotes when WAL stream with master is cut + delay master/slave

On 2013-01-17 13:47:41 +0900, Michael Paquier wrote:
> Hi all,
>
> There is a strange bug with the latest master head (commit 7fcbf6a).
> When the WAL stream with a master is cut on a slave, slave returns a FATAL
> (well normal...), but then enters in recovery process and automatically
> promotes.
> Here are more details about the logs on slave (I simply killed the master
> manually):
> FATAL: could not receive data from WAL stream:
> cp: cannot stat
> ‘/home/michael/bin/pgsql/archive/master/000000010000000000000004’: No such
> file or directory
> LOG: record with zero length at 0/401E1B8
> LOG: redo done at 0/401E178
> LOG: last completed transaction was at log time 2013-01-17
> 20:27:53.180971+09
> cp: cannot stat ‘/home/michael/bin/pgsql/archive/master/00000002.history’:
> No such file or directory
> LOG: selected new timeline ID: 2
> cp: cannot stat ‘/home/michael/bin/pgsql/archive/master/00000001.history’:
> No such file or directory
> LOG: archive recovery complete
> DEBUG: resetting unlogged relations: cleanup 0 init 1
> LOG: database system is ready to accept connections
> LOG: autovacuum launcher started
> DEBUG: archived transaction log file "000000010000000000000004"
> DEBUG: archived transaction log file "00000002.history"
> LOG: statement: create table bn (a int);
> DEBUG: autovacuum: processing database "postgres"
>
> Slave does not try anymore to reconnect to master with messages of the type:
> FATAL: could not connect to the primary server
>
> I also noticed that there is some delay until modifications on master are
> visible on slave.
> For example run a simple CREATE TABLE and the new table is not
>
> [some bisecting later...]
>
> I think that bug has been introduced by commit 7fcbf6a.
> Before splitting xlog reading as a separate facility things worked
> correctly.
> There are also no delay problems before this commit.

Ok, my inkling proved to be correct, its two related issues:

a ) The error handling in XLogReadRecord is inconsistent, it doesn't
always reset the necessary things.

b) ReadRecord: We cannot not break out of the retry loop in readRecord
just so, just removing the break seems correct.

c) ReadRecord: (minor): We should log an error even if errormsg is not
set, otherwise we wont jump out far enough.

I think at least a) and b) is the result of merges between development
of different people, sorry for that.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services