BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster

From: PG Bug reporting form <noreply(at)postgresql(dot)org>
To: pgsql-bugs(at)lists(dot)postgresql(dot)org
Cc: schefner20(at)gmail(dot)com
Subject: BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster
Date: 2026-11-16 14:08:50
Message-ID: 19908-3db6f3c4f2febdcf@postgresql.org
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

The following bug has been logged on the website:

Bug reference: 19908
Logged by: Nick Schefner
Email address: schefner20(at)gmail(dot)com
PostgreSQL version: 18.6
Operating system: RHEL Rocky Linux 9.8
Description:

We have the following setup:

- 2 Postgres Servers | One running as primary the other running as standby
- Our Instances are managed though podman quadlets with the following
settings:

postgresql.container
[Unit]
Description=Podman postgresql.container
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/tmp/podman-run-26/containers
StartLimitIntervalSec=400
StartLimitBurst=3

[Container]
Image=<host>/<images>:14.24-0-tsdb2.11.0-0-gis3.1.12-0
ContainerName=postgresql
Exec=/usr/pgsql-14/bin/postgres -D /postgres/<instance>/
PublishPort=0.0.0.0:****:****
Volume=/etc/ssl/postgres:/etc/ssl/postgres:ro
Volume=/var/lib/pgsql/for_podman/:/var/lib/pgsql/from_outside/:ro
Volume=/postgres/<instance>:/postgres/<instance>
Volume=/etc/repmgr/rdm_tsm:/etc/repmgr/rdm_tsm:ro
Environment=PGDATA=/postgres/<instance>
Environment=PGPORT=****
Environment=PGPASSFILE=/var/lib/pgsql/from_outside/.pgpass
ShmSize=20g
User=26:26
UserNS=keep-id
PodmanArgs=--add-host host.containers.internal:host-gateway
[Service]
Environment=REGISTRY_AUTH_FILE=/var/lib/pgsql/.config/containers/.auth.json
Restart=on-failure
RestartSec=10
ExecReload=/usr/bin/podman exec postgresql /bin/kill -HUP 1
[Install]
WantedBy=default.target

The data directories lie on an NFS share.

Now to the bug:

I wanted to upgrade the version 14 Instance to the 18.6 version (i sadly
just realized that you released 18.7 a few days ago).
Before the upgrade I wanted to test 'pg_upgrade --check ...' on the 14
Instance while it was still running, since --check signals to me that
nothing should break running it.
None the less the command broke my instance. Im going to provide the
commands i did now with some errors.

1. I created a one-time podman container for upgrading the instance
podman run -it --rm --name pg-upgrade --userns=keep-id --user root
\
-v /postgres:/postgres -v /etc/ssl/postgres:/etc/ssl/postgres:ro \
git.ufz.de:4567/it/databases/postgres/images:14.24-0-tsdb2.11.0-0-gis3.1.12-0
bash
2. I installed the new versions into the one-time podman container
dnf install -y postgresql18-server postgresql18-contrib
postgis36_18-3.6.4 repmgr_18
3. I initialized the version 18 Instance into the new directory. (still in
the container)
su postgres -c "/usr/pgsql-18/bin/initdb -D /postgres/rdm_tsm/18
--data-checksums --encoding=UTF-8 --locale=de_DE.utf8 --wal-segsize=512"
4. I revoked the old barman functions since they got renamed after Version
14 (on the 14 Instance)
REVOKE ALL ON FUNCTION pg_start_backup(text, boolean, boolean) FROM
barman;
REVOKE ALL ON FUNCTION pg_stop_backup() FROM barman;
REVOKE ALL ON FUNCTION pg_stop_backup(boolean, boolean) FROM barman;
5. I ran pg_upgrade --check inside the one-time container
cd /postgres/rdm_tsm
su postgres -c "/usr/pgsql-18/bin/pg_upgrade -b /usr/pgsql-14/bin -B
/usr/pgsql-18/bin -d /postgres/rdm_tsm -D /postgres/rdm_tsm/18 -p 50432 -P
50433 --check"

Here it failed with the following error:

Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions ok

could not open file "/postgres/rdm_tsm/postmaster.pid": No such file or
directory
Failure, exiting

Here might be some more errors helping you understand what happen:

/postgres/rdm_tsm/18/pg_upgrade_output.d/20261116T073424.501/log/pg_upgrade_server.log:
pg_upgrade run on Mon Nov 16 07:34:24 2026
command: "/usr/pgsql-14/bin/pg_ctl" -w -l ".../pg_upgrade_server.log" -D
"/postgres/rdm_tsm" -o "-p 50432 -b -c listen_addresses='' -c
unix_socket_permissions=0700" start
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-11-16 07:34:24 CET [253] @ INFO:
timescaledb loaded
2026-11-16 07:34:25 CET [253] @ LOG: redirecting log output to logging
collector process
. stopped waiting
pg_ctl: could not start server

live server log:
07:34:50 [1] @ LOG: performing immediate shutdown because data directory
lock file is invalid
07:34:50 ... FATAL: the database system is shutting down (pgbouncer,
monitoring, barman, repmgr)

And this after restart attempts:
PANIC: could not find redo location ... (WAL already deleted by "rm %p"
in the archive_command)
07:48:45 [3] @ FATAL: requested timeline 10 does not contain minimum
recovery point 48AB/7B318D48 on timeline 0
07:48:45 [1] @ LOG: aborting startup due to startup process failure
(The "rm %p" is probably wrong and resulted into the recovery failure)

Apperently the upgrade command couldn't correctly retrieve the
postmaster.pid file or thought it was stale and tried to start the server
anyway.
As a side effect it completely wiped the postmaster.pid and created a new
one breaking the instance.

Possible Explanation:
- pg_upgrade read the postmaster.pid and checked if the PID exists
- it then couldn't find a process with that PID because the container,
where pg_upgrade was running on, has its own PID namespace
- since it can't find the PID, pg_upgrade startet a second instance

Suggestion:
I've tried to figure out if this might be fixed in 18.7 but looking
through the commits of pg_upgrade.c I couldn't find one that solved the
problem.
A Solution could be to just not try to start the instance when a
postmaster.pid was found e.g:
In 'pg_upgrade.c' :

static void check_clusters_stopped(void) {
check_pghost_envvar();
if (pid_lock_file_exists(old_cluster.pgdata))
{
if (!user_opts.check)
pg_fatal("A postmaster.pid file exists in the old cluster
data directory \"%s\".\n"
"Shut down the postmaster servicing the old
cluster and try again.\n"
"If the file is stale, start and cleanly
stop the old server, or remove the file\n"
"once you are certain no server (in any
container or on any host) uses this directory.",
old_cluster.pgdata);
user_opts.live_check = true;
}

if (pid_lock_file_exists(new_cluster.pgdata))
pg_fatal("A postmaster.pid file exists in the new cluster data
directory \"%s\".\n"
"Shut down the postmaster servicing the new
cluster and try again.\n"
"If the file is stale, remove it once you are
certain no server uses this directory.",
new_cluster.pgdata);
}
I rewrote the function, rebuild via meson and ran the tests and all came
back as passed.
The drawback would be that stale postmaster.pid files cant be ignored.
So a different solution could be preferable.

Documentation:
The Docs of pg_upgrade don't mention this behaviour. It states that
pg_upgrade starts a short lived postmaster on both instances but doesnt
state the same for --check (where I expect that nothing can possibly break).
With this information I could have known that i need to provide the current
clusters --pid and --ipc variables when starting the one-time upgrade
container.
So I would maybe suggest a note in the documentation, describing this issue.

Thank you for you work and sorry if I wrote to much
Feel free to contact me if you need further configs etc.

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Nisha Moond 2026-11-16 15:15:40 Re: BUG #19892: Logical replication skips UPDATE/DELETE of rows with NULL in a hash-indexed subscriber column
Previous Message Joao Rito 2026-11-16 14:02:39 Re: BUG #19817: CREATE UNIQUE INDEX on a partitioned table can silently attach an invalid, not-ready child index, le