Re: [HACKERS] pg_upgrade to clusters with a different WAL segment size

From: "Bossart, Nathan" <bossartn(at)amazon(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>, Jeremy Schneider <schneider(at)ardentperf(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Michael Paquier <michael(dot)paquier(at)gmail(dot)com>, Andres Freund <andres(at)anarazel(dot)de>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [HACKERS] pg_upgrade to clusters with a different WAL segment size
Date: 2018-02-07 18:34:58
Message-ID: 57923032-5892-4454-BC00-CA69B695EAEE@amazon.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Apologies for the delay.

Here is a patch to allow users to change the WAL segment size of a cluster with
pg_resetwal. Like the '--wal-segize' option in initdb, the new '-s' option
accepts segment sizes in megabytes. This patch also fixes two small issues that
I noticed.

The first is a division-by-zero error caused when the control data must be
guessed. If the control data is damaged, WalSegSz will not be set to the
guessed WAL segment size, resulting in an error during XLogFromFileName(). The
attached patch resolves this issue by ensuring WalSegSz is set to a valid value
after either reading or guessing the control values.

The second issue is with the automatically determined new WAL starting address
calculation. Currently, the starting address is determined by looking at the
last WAL file name, rounding the WAL byte position up to the next segment of the
new size, and then incrementing to the next WAL segment for the new size. I
believe this can cause the WAL byte position to go backwards in some cases (e.g.
WAL position is towards the end of a 1024 MB segment and pg_resetwal is used to
switch the segment size to 1 MB). Since the "current" WAL byte position
calculated in pg_resetwal is always the beginning of the latest log, the current
strategy may not yield a byte position sufficiently far ahead. The attached
patch changes this logic to determine the last byte position in the latest log
(for the old WAL segment size) prior to aligning to the new segment size and
incrementing.

Note that some segment size changes will cause old WAL file names to be reused.
The new documentation for '-s' contains a note warning of this side effect. I
considered a couple of strategies to prevent this, but I chose not to include
any in the patch based on previous correspondence in this thread. If file name
overlap is an issue, users can already use the '-l' option to set a safe WAL
starting address.

Nathan

Attachment Content-Type Size
pg_resetwal_segsize_v1.patch application/octet-stream 4.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andres Freund 2018-02-07 18:41:37 Re: [HACKERS] [POC] Faster processing at Gather node
Previous Message Robert Haas 2018-02-07 18:11:06 Re: [HACKERS] Transactions involving multiple postgres foreign servers