BUG #19889: ShmemRequestHash underestimates storage for highly partitioned hashes

The following bug has been logged on the website:

Bug reference: 19889
Logged by: Faye
Email address: imchifan(at)mail(dot)com
PostgreSQL version: Unsupported/Unknown
Operating system: Linux/amd64
Description:

ShmemRequestHash() reserves too little contiguous memory when a small nelems
value is combined with HASH_PARTITION and a larger partition count. A
request for one element with 512 partitions fails during initialization with
"failed to initialize hash table", although the named shared-memory
allocation is present.

The request uses supported hash options and should initialize successfully.
The apparent cause is that hash_estimate_size(nelems, entrysize) does not
account for num_partitions, while init_htab() raises the bucket count to at
least num_partitions. Here the estimate covers one 256-bucket segment, but
initialization requires two.

How to repeat:

Save these four files in an empty directory. Set PG_CONFIG to pg_config from
the PostgreSQL installation under test if necessary, then run ./run.sh as a
non-root user permitted to install a module into that installation.

Makefile:

MODULES = partition_hash_probe
PGFILEDESC = "partition_hash_probe - shared hash sizing probe"

PG_CONFIG ?= pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

partition_hash_probe.c:

#include "postgres.h"

#include "fmgr.h"
#include "storage/shmem.h"
#include "utils/hsearch.h"

PG_MODULE_MAGIC;

typedef struct PartitionHashEntry
{
uint32 key;
uint32 value;
} PartitionHashEntry;

static HTAB *partition_hash;

static void
request_partition_hash(void *arg)
{
ShmemRequestHash(.name = "partition_hash_probe",
.nelems = 1,
.ptr = &partition_hash,
.hash_info.keysize = sizeof(uint32),
.hash_info.entrysize = sizeof(PartitionHashEntry),
.hash_info.num_partitions = 512,
.hash_flags = HASH_ELEM | HASH_BLOBS |
HASH_PARTITION,
);
}

static const ShmemCallbacks callbacks = {
.request_fn = request_partition_hash,
.flags = SHMEM_CALLBACKS_ALLOW_AFTER_STARTUP,
};

void
_PG_init(void)
{
RegisterShmemCallbacks(&callbacks);
}

test.sql:

\set ON_ERROR_STOP on

LOAD 'partition_hash_probe';

SELECT name, size > 0 AS allocated
FROM pg_shmem_allocations
WHERE name = 'partition_hash_probe';

run.sh:

#!/bin/sh
set -eu
: "${PG_CONFIG:=pg_config}"
bindir=$($PG_CONFIG --bindir)
work=$(mktemp -d "${TMPDIR:-/tmp}/partition-hash.XXXXXX")
trap '"$bindir/pg_ctl" -D "$work/data" -m immediate stop >/dev/null 2>&1
|| true' EXIT HUP INT TERM
make PG_CONFIG="$PG_CONFIG"
make PG_CONFIG="$PG_CONFIG" install
"$bindir/initdb" -D "$work/data" --no-locale --auth=trust >/dev/null
"$bindir/pg_ctl" -D "$work/data" -l "$work/server.log" -o "-k $work -c
listen_addresses=''" start >/dev/null
if "$bindir/psql" -h "$work" -d postgres -X -v ON_ERROR_STOP=1 -f
test.sql >"$work/result.out" 2>&1; then
echo "NOT REPRODUCED: LOAD succeeded"
exit 1
fi
cat "$work/result.out"
grep -q 'failed to initialize hash table.*partition_hash_probe'
"$work/result.out"
echo "BUG REPRODUCED: estimator-sized partitioned hash failed
initialization"

The confirmed execution used a standalone backend. With the identifiers
above, its relevant output is:

ERROR: failed to initialize hash table "partition_hash_probe"

The subsequent pg_shmem_allocations query showed the allocation left by the
failed initialization:

name | allocated
----------------------+-----------
partition_hash_probe | t

The supplied run.sh checks for the same initialization error and, when it
finds it, prints:

BUG REPRODUCED: estimator-sized partitioned hash failed initialization

That line is the runner's oracle; it was not part of the standalone-backend
output. On a corrected build, LOAD should succeed and the query should
return allocated = t. A failed after-startup initialization can leave an
unusable ShmemIndex entry and its static allocation behind, so the
disposable server should be restarted or removed after the test.

The issue was reproduced on PostgreSQL 20devel on Linux/amd64, compiled with
GCC 11.4.0. PostgreSQL 18.6 predates the target shared-memory API and was
not used to exercise this module. This occurs when a very small shared hash
is requested with enough partitions to raise its initialization-time bucket
minimum above the storage estimate.