Re: pg_rewind test race condition..?

On 04/28/2015 11:02 AM, Stephen Frost wrote:
> Heikki,
>
> Not sure if anyone else is seeing this, but I'm getting regression
> test failures when running the pg_rewind tests pretty consistently
> with 'make check'. Specifically with "basic remote", I'm getting:
>
> source and target cluster are on the same timeline
> Failure, exiting
>
> in regress_log/pg_rewind_log_basic_remote.
>
> If I throw a "sleep(5);" into t/001_basic.pl before the call to
> RewindTest::run_pg_rewind($test_mode); then everything works fine.

The problem seems to be that when the standby is promoted, it's a
so-called "fast promotion", where it writes an end-of-recovery record
and starts accepting queries before creating a real checkpoint.
pg_rewind looks at the TLI in the latest checkpoint, as it's in the
control file, but that isn't updated until the checkpoint completes. I
don't see it on my laptop normally, but I can reproduce it if I insert a
"sleep(5)" in StartupXLog, just before it requests the checkpoint:

--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7173,7 +7173,10 @@ StartupXLOG(void)
* than is appropriate now that we're not in standby mode anymore.
*/
if (fast_promoted)
+ {
+ sleep(5);
RequestCheckpoint(CHECKPOINT_FORCE);
+ }
}

The simplest fix would be to force a checkpoint in the regression test,
before running pg_rewind. It's a bit of a cop out, since you'd still get
the same issue when you tried to do the same thing in the real world. It
should be rare in practice - you'd not normally run pg_rewind
immediately after promoting the standby - but a better error message at
least would be nice..
- Heikki