[bug fix] pg_rewind creates corrupt WAL files, and the standby cannot catch up the primary

From: "Tsunakawa, Takayuki" <tsunakawa(dot)takay(at)jp(dot)fujitsu(dot)com>
To: "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: [bug fix] pg_rewind creates corrupt WAL files, and the standby cannot catch up the primary
Date: 2018-03-01 01:26:32
Message-ID: 0A3221C70F24FB45833433255569204D1F8DAAA2@G01JPEXMBYT05
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


Our customer hit another bug of pg_rewind with PG 9.5. The attached patch fixes this.


After a long run of successful pg_rewind, the synchronized standby could not catch up the primary forever, emitting the following message repeatedly:

LOG: XX000: could not read from log segment 000000060000028A00000031, offset 16384: No error


If the primary removes WAL files that pg_rewind is going to get, pg_rewind leaves 0-byte WAL files in the target directory here:

/* Truncate the old file out of the way, if any */
open_target_file(entry->path, true);
fetch_file_range(entry->path, 0, entry->newsize);

pg_rewind completes successfully, create recovery.conf, and then start the standby in the target cluster. walreceiver receives WAL records and write them to the 0-byte WAL files. Finally, xlog reader complains that he cannot read a WAL page.


pg_rewind deletes the file when it finds that the primary has deleted it.


BTW, should pg_rewind really copy WAL files from the primary? If the sole purpose of pg_rewind is to recover an instance to use as a standby, can pg_rewind just remove all WAL files in the target directory, because the standby can get WAL files from the primary and/or archive?

Related to this, shouldn't pg_rewind avoid copying more files and directories like pg_basebackup? Currently, pg_rewind doesn't copy postmaster.pid, postmaster.opts, and temporary files/directories (pg_sql_tmp/).

Takayuki Tsunakawa

Attachment Content-Type Size
pg_rewind_corrupt_wal.patch application/octet-stream 2.2 KB


Browse pgsql-hackers by date

  From Date Subject
Next Message Amit Langote 2018-03-01 01:27:54 Re: [HACKERS] path toward faster partition pruning
Previous Message Craig Ringer 2018-03-01 01:16:14 Re: Online enabling of checksums