| From: | Nikolay Samokhvalov <nik(at)postgres(dot)ai> |
|---|---|
| To: | pgsql-hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org> |
| Subject: | Prevent premature startup of pg_upgrade targets |
| Date: | 2026-08-14 05:31:59 |
| Message-ID: | CAM527d_MAVFE8RVM_Qr8-P4opUSxemY2r=MeEiV+odTu6dw4cw@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi hackers,
While investigating several corruption cases after major-version
upgrades, I noticed that a pg_upgrade target can be started normally
before pg_upgrade finishes, regardless of transfer mode.
On current master, setup() checks postmaster.pid for both clusters,
and get_control_data() checks for a clean shutdown. Both are initial
checks. There is no startup interlock during the upgrade.
pg_upgrade later starts and stops its own postmasters, so an external
startup may make a later start fail. But offline phases such as file
transfer, pg_resetwal, and syncing remain unprotected.
A premature startup allows clients, autovacuum, and background workers
to write WAL and advance xid and MultiXact state while pg_upgrade is
changing the cluster. pg_upgrade may still complete successfully.
In one production case, pg_upgrade completed successfully in copy
mode, and corruption was noticed weeks later. We observed missing
pg_xact state for tuple xmin, missing TOAST chunks, visibility map
inconsistencies, and VACUUM failing with “uncommitted xmin ... needs
to be frozen”. We also found evidence that the target had been started
during the upgrade. We have not proven that this caused the
corruption, but allowing such a startup looks unsafe. Refusing it
would make pg_upgrade more robust against automation errors.
There is already narrower protection for the old cluster. In link and
swap modes, disable_old_cluster() renames pg_control before file
transfer. The target has no equivalent protection.
I propose a durable marker in the target data directory that prevents
normal startup. Postmasters started by pg_upgrade would bypass it. The
marker would be removed and fsynced only when startup is safe.
For standby upgrades using rsync, the marker may need to remain after
pg_upgrade completes and be removed explicitly after rsync.
Some related discussions:
- old-cluster protection:
https://www.postgresql.org/message-id/flat/20120228162106.GA20764%40momjian.us
- writes during binary upgrade:
https://www.postgresql.org/message-id/flat/20210121152357.s6eflhqyh4g5e6dv%40dalibo.com
- logical replication launcher during pg_upgrade:
https://www.postgresql.org/message-id/E1qyQMD-003XXN-Mf%40gemulon.postgresql.org
- standby upgrades using rsync:
https://www.postgresql.org/message-id/flat/CAM527d8heqkjG5VrvjU3Xjsqxg41ufUyabD9QZccdAxnpbRH-Q%40mail.gmail.com
What do you think?
regards,
Nik
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Japin Li | 2026-08-14 05:32:24 | Re: Compression of bigger WAL records |
| Previous Message | Hayato Kuroda (Fujitsu) | 2026-08-14 05:29:22 | RE: [PATCH] Preserve replication origin OIDs in pg_upgrade |