Pg_rewind cannot load history wal

From: Richard Schmidt <Richard(dot)Schmidt(at)metservice(dot)com>
To: "pgsql-general(at)lists(dot)postgresql(dot)org" <pgsql-general(at)lists(dot)postgresql(dot)org>
Subject: Pg_rewind cannot load history wal
Date: 2018-08-01 21:09:30
Message-ID: HK0PR01MB200359DEA9A85774C9E3A799912D0@HK0PR01MB2003.apcprd01.prod.exchangelabs.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

We have been struggling to get pr_rewind to work. In desperation we have been trying to make the use case as simple as possible:)

We have two databases servers running Postgres 10 on two different machine in the normal Primary/Standby configuration.
Both machines write their WAL archive logs to the same shared drive (called /ice_dev/wal_archive).
The configuration has the following terms
archive_mode = always
archive_command = 'test ! -f /ice-dev/wal_archive/%f && cp %p /ice-dev/wal_archive/%f'
full_page_writes = on
wal_log_hints = on
Checksums are enabled

Our procedure that runs on machine A and B is as follows:

1. Build new databases on A and B, and configure A as Primary and B as Standby databases.
2. Make some changes to the A (the primary) and check that they are replicated to the B (the standby)
3. Promote B to be the new primary
4. Switch of the A (the original primary)
5. Add the replication slot to B (the new primary) for A (soon to be standby)
6. Add a recovery.conf to A (soon to be standby). File contains recovery_target_timeline = 'latest' and restore_command = 'cp /ice-dev/wal_archive/%f "%p"
7. Run pg_rewind on A - this appears to work as it returns the message 'source and target cluster are on the same timeline no rewind required';
8. Start up server A (now a slave)

At this point A is in a read only mode but not replicating. Its logs contain the following repeating message

2018-08-01 20:30:58 UTC [7257]: [1] user=,db=,app=,client= FATAL: could not start WAL streaming: ERROR: requested starting point 0/6000000 on timeline 1 is not in this server's history
DETAIL: This server's history forked from timeline 1 at 0/57639D0.
cp: cannot stat '/ice-dev/wal_archive/00000002.history': No such file or directory
cp: cannot stat '/ice-dev/wal_archive/00000003.history': No such file or directory
cp: cannot stat '/ice-dev/wal_archive/00000002.history': No such file or directory
2018-08-01 20:30:58 UTC [6840]: [48] user=,db=,app=,client= LOG: new timeline 2 forked off current database system timeline 1 before current recovery point 0/6000098
cp: cannot stat '/ice-dev/wal_archive/000000010000000000000006': No such file or directory

We can see the 00000002.history file in B's wal directory.....but it never appears in the wal_archive directory - not even if we issue a checkout or even restart the server.
00000003.history does not appear to exist on either of the machines.

Any ideas what we are doing wrong?
Thanks. Richard

This email and any attachments may contain confidential information. If you are not the intended recipient, your use or communication of the information is strictly prohibited. If you have received this message in error please notify MetService immediately.

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Igor Korot 2018-08-02 01:08:19 Problem building libpq
Previous Message Stephen Eilert 2018-08-01 20:08:37 Re: Template0 datfrozenxid age is 160million and progressing