Re: BUG #19764: ManualBug3 Assert failure in pgstat_setup_shared_refs crashes initdb post-bootstrap subprocess

On 10/15/26 18:34, Tomas Vondra wrote:
>
>
> On 10/15/26 17:05, PG Bug reporting form wrote:
>> The following bug has been logged on the website:
>>
>> Bug reference: 19764
>> Logged by: Yuelin Wang
>> Email address: 1217816127(at)qq(dot)com
>> PostgreSQL version: 19rc1
>> Operating system: Linux (Ubuntu 24.04, x86_64)
>> Description:
>>
>> ### What happens?
>>
>> During `initdb`, the "performing post-bootstrap initialization" subprocess
>> calls `pgstat_initialize()` (which sets `pgStatLocal.shmem->gc_request_count
>> = 1` via `StatsShmemInit`), then calls `StartupXLOG()`.
>> Inside `StartupXLOG()`, `pgstat_restore_stats()` is invoked.
>> After `pgstat_restore_stats()` completes, `gc_request_count` has been zeroed
>> out to 0 (verified via debug instrumentation in the working-tree build).
>> Later, during catalog initialization (`InitPostgres` calls
>> `table_beginscan_catalog`, which calls `heap_beginscan`, which calls
>> `pgstat_assoc_relation`), `pgstat_get_entry_ref` is invoked for the first
>> time in this subprocess.
>> This triggers `pgstat_setup_shared_refs()`
>> (src/backend/utils/activity/pgstat_shmem.c), which reads `pgStatSharedRefAge
>> = pg_atomic_read_u64(&pgStatLocal.shmem->gc_request_count)` and gets 0.
>> Immediately after, the function asserts `Assert(pgStatSharedRefAge != 0)` at
>> pgstat_shmem.c:939, which fires and aborts the subprocess, causing initdb to
>> fail and clean up the data directory.
>> The violated invariant is: `gc_request_count` must remain at its initialized
>> value (1) and never fall to 0, since `pgstat_setup_shared_refs` relies on a
>> non-zero value to guarantee `pgStatSharedRefAge != 0`.
>> The `own_hash` feature introduced in commit 44539786b8 added `entry_counts`,
>> `kind_hash_handles`, and `raw_kind_dsa_area` arrays to
>> `PgStat_ShmemControl`, altering the struct layout and the initialization
>> sequence in `pgstat_restore_stats`/`pgstat_read_statsfile`.
>> The debug trace confirms that `gc_request_count` transitions from 1 to 0
>> precisely during `pgstat_restore_stats()`.
>> This matches the Entry91 root-cause pattern: a field that must be
>> initialized to a sentinel non-zero value before use is inadvertently reset
>> to the uninitialized zero value by a code path that was modified by the same
>> recent commit.
>>
>> ### To Reproduce
>>
>> Start a fresh PostgreSQL ASAN instance with initdb:
>>
>> ```sh
>> TMPDIR=$(mktemp -d /tmp/pg_scan_XXXX)
>> DATADIR=$TMPDIR/pgdata
>> PGUSER=$(whoami)
>>
>> ASAN_OPTIONS=detect_leaks=0:abort_on_error=0:log_path=$TMPDIR/asan \
>> /data/ylwang/Projects/DirectedDB/latesttarget/postgre/build_asan/install/bin/initdb
>> \
>> -D $DATADIR -U $PGUSER --no-locale --encoding=UTF8 2>&1
>> ```
>>
>> Trigger output:
>>
>> ```
>> The files belonging to this database system will be owned by user
>> "yuelinwang".
>> This user must also own the server process.
>>
>> The database cluster will be initialized with locale "C".
>> The default text search configuration will be set to "english".
>>
>> Data page checksums are enabled.
>>
>> creating directory .../pgdata ... ok
>> creating subdirectories ... ok
>> selecting dynamic shared memory implementation ... posix
>> selecting default "max_connections" ... 100
>> selecting default "shared_buffers" ... 128MB
>> selecting default time zone ... Asia/Singapore
>> creating configuration files ... ok
>> running bootstrap script ... ok
>> performing post-bootstrap initialization ... TRAP: failed
>> Assert("pgStatSharedRefAge != 0"), File: "pgstat_shmem.c", Line: 939, PID:
>> 3146498
>> /lib/x86_64-linux-gnu/libasan.so.8(+0x831e0)[...]
>> .../postgres(ExceptionalCondition+0x165)[...]
>> .../postgres(+0x14c282a)[...]
>> .../postgres(pgstat_get_entry_ref+0x33b)[...]
>> .../postgres(pgstat_prep_pending_entry+0x37)[...]
>> .../postgres(+0x14b9ee9)[...]
>> .../postgres(pgstat_assoc_relation+0x1f7)[...]
>> .../postgres(+0x5a3a24)[...]
>> .../postgres(heap_beginscan+0x8c0)[...]
>> .../postgres(+0x6e832b)[...]
>> .../postgres(table_beginscan_catalog+0x9a)[...]
>> .../postgres(+0x1963b82)[...]
>> .../postgres(InitPostgres+0x63a)[...]
>> .../postgres(PostgresMain+0x553)[...]
>> .../postgres(main+0x5d5)[...]
>> Aborted (core dumped)
>> child process exited with exit code 134
>> initdb: removing data directory ".../pgdata"
>> ```
>>
>> The assertion failure prevents initdb from completing and the data directory
>> is cleaned up automatically.
>>
>
> I can't reproduce this. Please provide step-by-step instructions how to
> trigger the issue.
>
> I tried this:
>
> CC=clang ./configure --prefix=/home/user/builds/asan \
> --enable-debug --enable-cassert \
> CFLAGS="-O0 -g -fno-omit-frame-pointer -fsanitize=address,undefined \
> -fno-sanitize=function -fno-sanitize-recover=all" \
> LDFLAGS="-fsanitize=address,undefined"
>
> and then after build
>
> export ASAN_OPTIONS=detect_leaks=0:abort_on_error=0
> initdb tmp/data2 --no-locale --encoding=UTF8 -U test
>
> No crash / assert failure. Furthermore, I fail to understand why would
> this require ASAN build, if it crashes on an assert.
>

I've done some more experiments, and I strongly suspect the AI tool
you're relying on simply works on a broken build. If I checkout and
build at 44539786b8^, and then update and do an incremental build at
44539786b8, I get very similar crashes:

----
TRAP: failed Assert("pgStatSharedRefAge != 0"), File:
"/tmp/pgstat_stale_tKJQ/postgres/build/../src/backend/utils/activity/pgstat_shmem.c",
Line: 505, PID: 498693
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(ExceptionalCondition+0x9e)
[0xbff043]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres() [0xa4660c]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(pgstat_get_entry_ref+0xc0)
[0xa469a0]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(pgstat_create_transactional+0x2e)
[0xa495ca]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(pgstat_create_relation+0x57)
[0xa42139]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(heap_create+0x2f1)
[0x5476d5]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(boot_yyparse+0x91a)
[0x531572]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(BootstrapModeMain+0x5e5)
[0x535cbb]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(main+0x2eb)
[0x7a353c]
/lib64/libc.so.6(+0x35b5) [0x79ff8c2d45b5]
----

Your report has a slightly different backtrace, but it also says Line
939, and that line does not have that assert at all (there's a different
assert on line 938).

Thus my theory is - this is a broken incremental build, due to the AI
thing not using --enable-depend (or doing make clean). Chances are the
backtraces differ because of that too. And if that assessment is
correct, it means the whole elaborate "What happens?" section is likely
just a pile of hallucinated garbage.

I may be wrong, of course. An easy way to prove that is by providing a
working reproducer. (But I find that unlikely - the chance of us not
noticing a crash in initdb, which *every* test run does multiple times,
is rather low IMO.)

regards

--
Tomas Vondra