Low frequency AIO checksum corruption on buildfarm member fruitcrow / GNU Hurd

Hi,

Every few days/weeks, a run on my Debian GNU/HUrd buildfarm member
fruitrow fails with something like this, most recently in [1], other
occurances are [2-9]:

|echo "# +++ regress check in contrib/pg_stat_statements +++" && PATH="/home/demo/client-code-REL_19_1/buildroot/REL_19_STABLE/pgsql.build/tmp_install/home/demo/build-farm-19.1/buildroot/REL_19_STABLE/inst/bin:/home/demo/client-code-REL_19_1/buildroot/REL_19_STABLE/pgsql.build/contrib/pg_stat_statements:$PATH" LD_LIBRARY_PATH="/home/demo/client-code-REL_19_1/buildroot/REL_19_STABLE/pgsql.build/tmp_install/home/demo/build-farm-19.1/buildroot/REL_19_STABLE/inst/lib:$LD_LIBRARY_PATH" INITDB_TEMPLATE='/home/demo/client-code-REL_19_1/buildroot/REL_19_STABLE/pgsql.build'/tmp_install/initdb-template ../../src/test/regress/pg_regress --temp-instance=./tmp_check --inputdir=. --bindir= --temp-config=/home/demo/build-farm-19.1/buildroot/tmp/buildfarm-JwzW2U/bfextra.conf --port=5798 --temp-config ../../contrib/pg_stat_statements/pg_stat_statements.conf --dbname=contrib_regression select dml cursors utility level_tracking planning user_activity wal entry_timestamp privileges extended parallel plancache squashing cleanup oldextversions
|# +++ regress check in contrib/pg_stat_statements +++
|# initializing database system by copying initdb template
|# using temp instance on port 5798 with PID 19255
|ERROR: invalid page in block 97 of relation "base/4/1255"
|ERROR: database "contrib_regression" does not exist
|# command failed: "psql" -X -q -c "CREATE DATABASE \\"contrib_regression\\" TEMPLATE=template0" -c "ALTER DATABASE \\"contrib_regression\\" SET lc_messages TO 'C';ALTER DATABASE \\"contrib_regression\\" SET lc_monetary TO 'C';ALTER DATABASE \\"contrib_regression\\" SET lc_numeric TO 'C';ALTER DATABASE \\"contrib_regression\\" SET lc_time TO 'C';ALTER DATABASE \\"contrib_regression\\" SET bytea_output TO 'hex';ALTER DATABASE \\"contrib_regression\\" SET timezone_abbreviations TO 'Default';" "postgres"
|Bail out!make: *** [../../src/makefiles/pgxs.mk:451: check] Fehler 2

The postmaster log for the pg_stat_statements regress check has this:

|2026-10-06 22:29:40.701 BST [19267:4] pg_regress LOG: statement: CREATE DATABASE "contrib_regression" TEMPLATE=template0
|2026-10-06 22:29:41.599 BST [19256:1] LOG: page verification failed, calculated checksum 48592 but expected 64049
|2026-10-06 22:29:41.599 BST [19256:2] CONTEXT: I/O worker executing I/O on behalf of process 19267
|2026-10-06 22:29:41.600 BST [19256:3] LOG: invalid page in block 97 of relation "base/4/1255"
|2026-10-06 22:29:41.600 BST [19256:4] CONTEXT: I/O worker executing I/O on behalf of process 19267
|2026-10-06 22:29:41.674 BST [19267:5] pg_regress ERROR: invalid page in block 97 of relation "base/4/1255"
|2026-10-06 22:29:41.674 BST [19267:6] pg_regress STATEMENT: CREATE DATABASE "contrib_regression" TEMPLATE=template0
|2026-10-06 22:29:41.717 BST [19267:7] pg_regress LOG: statement: ALTER DATABASE "contrib_regression" SET lc_messages TO 'C';ALTER DATABASE "contrib_regression" SET lc_monetary TO 'C';ALTER DATABASE "contrib_regression" SET lc_numeric TO 'C';ALTER DATABASE "contrib_regression" SET lc_time TO 'C';ALTER DATABASE "contrib_regression" SET bytea_output TO 'hex';ALTER DATABASE "contrib_regression" SET timezone_abbreviations TO 'Default';
|2026-10-06 22:29:41.718 BST [19267:8] pg_regress ERROR: database "contrib_regression" does not exist
|2026-10-06 22:29:41.718 BST [19267:9] pg_regress STATEMENT: ALTER DATABASE "contrib_regression" SET lc_messages TO 'C';ALTER DATABASE "contrib_regression" SET lc_monetary TO 'C';ALTER DATABASE "contrib_regression" SET lc_numeric TO 'C';ALTER DATABASE "contrib_regression" SET lc_time TO 'C';ALTER DATABASE "contrib_regression" SET bytea_output TO 'hex';ALTER DATABASE "contrib_regression" SET timezone_abbreviations TO 'Default';
|2026-10-06 22:29:41.724 BST [19267:10] pg_regress LOG: disconnection: session time: 0:00:01.026 user=demo database=postgres host=[local]
|2026-10-06 22:29:41.734 BST [19255:4] LOG: received fast shutdown request
|2026-10-06 22:29:41.734 BST [19255:5] LOG: aborting any active transactions
|2026-10-06 22:29:41.749 BST [19255:6] LOG: background worker "logical replication launcher" (PID 19263) exited with exit code 1
|2026-10-06 22:29:41.750 BST [19258:1] LOG: shutting down
|2026-10-06 22:29:41.751 BST [19258:2] LOG: checkpoint starting: shutdown fast
|2026-10-06 22:29:41.761 BST [19258:3] LOG: checkpoint complete: shutdown fast: wrote 3 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.006 s, sync=0.001 s, total=0.010 s; sync files=0, longest=0.000 s, average=0.000 s; distance=3085 kB, estimate=3085 kB; lsn=0/01A5FE00, redo lsn=0/01A5FE00
|2026-10-06 22:29:41.795 BST [19255:7] LOG: database system is shut down

The frequency is low, but so far it only ever hit in 18/19/HEAD and it
always had the 'CONTEXT: I/O worker executing I/O on behalf of process
[...]' message, which indicates this only happens in AIO.

I since configured the buildfarm member to keep the failing inst/build
tree, but when I run pg_checksums on it, there is no error, so I think
this is not storage corruption, but rather something in-memory:

|$ LANG=C /home/demo/build-farm-19.1/buildroot/REL_19_STABLE/instkeep.2026-10-06_22-14-32/bin/pg_checksums -v -D /home/demo/build-farm-19.1/buildroot/REL_19_STABLE/pgsqlkeep.2026-10-06_22-14-32/contrib/pg_stat_statements/tmp_check/data/ 2>&1 | grep 4.1255.$
|pg_checksums: checksums verified in file "/home/demo/build-farm-19.1/buildroot/REL_19_STABLE/pgsqlkeep.2026-10-06_22-14-32/contrib/pg_stat_statements/tmp_check/data//base/4/1255"

I tried to reproduce the problem, but so far had no luck.

Looking at
https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures (and
CC'ing Alexander), it looks like this error has not appeared on any
other buildfarm members, so it looks like it is specific to GNU Hurd.

Anybody have an idea on how to diagnose that and whether this is
interesting to pursue anyway?

Michael

[1] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-10-06%2021%3A14%3A32&stg=misc-check
[2] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-09-22%2013%3A10%3A04&stg=misc-check
[3] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-09-19%2001%3A10%3A03&stg=check
[4] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-09-04%2007%3A29%3A28&stg=misc-check
[5] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-08-19%2003%3A20%3A17&stg=check
[6] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-08-01%2013%3A10%3A03&stg=misc-check
[7] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-07-28%2003%3A24%3A18&stg=misc-check
[8] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-07-01%2014%3A56%3A27&stg=misc-check
[9] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-06-03%2002%3A48%3A51&stg=misc-check