Re: Adding basic NUMA awareness

Attached is v20261011, a reordered version of the NUMA series with my
clock-sweep changes merged into it rather than stacked on the end.

The previous version I posted put the whole partitioning series first and my
patches after it, which meant patches 4 through 8 talked about partitioning
a sweep that patch 9 then changed underneath them. Reading it back, the
ordering was telling the story backwards. So:

0001 shmem_populate and shmem_interleave GUCs (Tomas)
0002 infrastructure for partitioning of shared buffers (Tomas)
0003 NUMA: shared buffers partitioning (Tomas)
0004 clock-sweep: track one second chance per buffer, not a 0..5 count
0005 clock-sweep: admit demand-loaded pages on probation
0006 clock-sweep: bound the work one allocation can do
0007 clock-sweep: let the background writer spend chances when the sweep
starves
0008 clock-sweep: basic partitioning (Tomas)
0009 clock-sweep: balancing of allocations (Tomas)
0010 clock-sweep: scan all partitions (Tomas)
0011 partitioned clocksweep and NUMA goodies (Jakub)
0012 cached CPU/NUMA node and locality-aware balancing (Jakub)

At a high level:
0001-0003 lay out the memory,
0004-0007 settle how the sweep decides what to evict, and
0008-0012 partition that sweep across nodes.

If I had to group them:

0004-0007 - The changes don't represent a departure from the clock sweep
algorithm. The hand is untouched: ClockSweepTick(),
nextVictimBuffer and completePasses are exactly as they are
today, and a buffer that gets accessed between two visits is
still passed over. What changes is the bookkeeping the hand
consults.

0008-0012 - These are Tomas's or Jakub's patchs with the same subject
and authorship as before; the only changes are the
mechanical ones needed because the state they partition is
now one bit rather than a 0..5 counter.

And now more deatils:

0004: I replace the 0..5 usage counter with a single second-chance bit.
A pin sets it, a visit clears it, and the sweep reclaims a buffer
on the visit that finds it already clear. The counter's range was
always a trade-off, a large value approximates LRU, but "it can
take as many as BM_MAX_USAGE_COUNT+1 complete cycles of the
clock-sweep hand to find a free buffer". The extra resolution
only distinguishes buffers by how many times they were touched
between passes of the hand, which on a large pool is a long and
load-dependent interval; what it reliably buys is up to five
revolutions of scanning before a hot buffer can be reclaimed. One
bit keeps the property that matters and makes the work needed to
find a victim independent of how hot the pool has been. So I
narrowed the field. BUF_USAGECOUNT_BITS/_SHIFT/_MASK/_ONE,
BUF_STATE_GET_USAGECOUNT and BM_MAX_USAGE_COUNT all go away. The
state becomes a buffer flag in the bit position the counter
occupied which reclaims four bits of the state word and lets the
existing flag machinery carry it, skinnying up buf_internals.h.

0005: This changes admission into the pool, a demand-loaded page arrives
without its chance, a second access promotes it. A one-touch scan
then fills and drains the pool without evicting pages that are
being re-read. This is a separate commit because it is a separate
decision. I took care in 0004 to keep the pre-existing admission
policy exactly for clarity and to demonstrate the hit-ratio costs
if any exist.

0006: This counts the chances one StrategyGetBuffer() call has spent
and, past a threshold, takes the next unpinned buffer instead of
spending another, which bounds the work a single allocation can
do.

0007: And this commit enables the background writer to "spend chances"
as it scans, but only while that claim rate is rising so as to
avoid brownouts under heavy load.

pg_buffercache keeps its user-visible shape throughout. usagecount and
pg_buffercache_usage_counts() are documented API; they keep their names and
types and now report 0 or 1.

Portability fixes.

0001 used MAP_POPULATE unguarded, which is Linux-only, so the series did
not compile on FreeBSD at all. I hit this building on v14.5. It is now
under #ifdef MAP_POPULATE.

0010 had a bare #include <numa.h> in postgres.c while the code using it
was already under USE_LIBNUMA; the include is now guarded the same way.
Both would have shown up on the non-Linux buildfarm animals.

Build fixes.

0002 bumps pg_buffercache to 1.8 and adds the new 1.7--1.8 script to the
Makefile but not to meson.build, so a meson build installs no upgrade
script and CREATE EXTENSION pg_buffercache fails;
contrib/pg_buffercache/regress fails with it.

0008 uses lookup_type_cache() without including utils/typcache.h. And
0007's assert on the computed victim index compares a uint32 >= 0, which
-Werror=type-limits rejects. All three are folded into the patches that
introduced them.

The claim threshold in 0005 is 128, where the version I posted had a
cache-line-derived 32. That was a placeholder picked before I measured the
trade-off. The worst case an allocation reaches is almost exactly the
threshold plus two, so the tail is a direct function of it; the cost is
evicting a buffer that had not finished probation. On a workload whose hot
set most queries re-read, a threshold of 4 leaves 4,519 hot pages
non-resident and costs 1.8 points of hot-set hit ratio, 32 leaves 11 pages
and costs 0.03 points, and at 128 nothing hot is displaced and the cost is
not measurable. I also tried making it adaptive, since that is the obvious
suggestion, and it was worse than a fixed value on both axes.

Benchmark numbers.

I measured three arms:
stock 20devel,
stock plus the NUMA series alone
and stock plus this series

at identical revisions on two bare-metal hosts, so the replacement-rule
effect can be separated from the hardware:

x86_64 z1d.metal, 48 vCPU, two NUMA nodes, 429k random read IOPS
aarch64 m7gd.metal, 64 vCPU, one NUMA node, 663k random read IOPS

Both Fedora 44, XFS on a RAID-0 of instance-local NVMe, huge pages on,
THP and NUMA balancing off, zipfian s=1.10, working set 1.15x the pool,
24 clients, three runs, medians. "tail" is the most clock advances any
single allocation performed. "adv/victim" the average number of ticks
before finding a victim. "tps" transactions per second. "hit%" the
cache hit ratio.

x86_64 tps adv/victim tail hit%
4GB 20devel 664,155 7.57 9,552 98.9997
4GB + NUMA series 657,125 7.51 43,594 98.9999
4GB + this series 660,104 3.97 442 98.9628
8GB 20devel 661,967 7.48 7,696 99.0679
8GB + NUMA series 659,621 7.14 35,006 99.0544
8GB + this series 659,291 4.05 523 99.0431
32GB 20devel 658,963 6.18 12,378 99.1391
32GB + NUMA series 655,628 7.54 51,287 99.1386
32GB + this series 657,134 4.26 466 99.1544

aarch64 tps adv/victim tail hit%
4GB 20devel 1,339,205 7.47 10,267 99.0074
4GB + NUMA series 1,335,287 7.51 28,614 99.0071
4GB + this series 1,325,400 4.00 130 98.9690
8GB 20devel 1,339,729 7.54 6,936 99.0826
8GB + NUMA series 1,332,664 7.46 26,852 99.0833
8GB + this series 1,325,256 4.00 130 99.0484
32GB 20devel 1,332,376 7.58 4,946 99.2054
32GB + NUMA series 1,328,510 7.16 20,841 99.1948
32GB + this series 1,315,824 4.02 129 99.1926

The two axes move for different reasons and both results hold on both
architectures. Advances per victim fall from about 7.5 to about 4.0 (a
1.45-1.91x reduction on x86_64 and 1.87-1.89x on aarch64) which is the
second-chance bit plus probationary admission, and is as close to
architecture-independent as I could ask for. The worst single
allocation falls to 442-523 advances on x86_64 and to 129-130 on
aarch64, 14-26x and 38-78x better than stock. On aarch64 that is the
threshold plus two almost exactly, and flat to 1.01x across an eightfold
pool range where stock varies by 2.08x; on x86_64 the bound varies 1.18x
against stock's 1.61x.

Throughput costs -0.28% to -0.61% on x86_64 and -1.03% to -1.24% on aarch64.
Hit ratio stays within 0.04 points everywhere, and is 0.015 points better at
32GB on x86_64. The bound engages 75 to 420 times in a window serving some
40 to 80 million queries, which is the behavior I want: a ceiling on the
unlucky allocations rather than a continuous change in policy.

One result I did not expect was the NUMA series on its own makes the
tail 4.1-4.6x worse than unpartitioned master on x86_64, and 2.8-4.2x
worse on aarch64. That follows from the design, each backend sweeps its
own share of the pool, so its partition can run out of candidates while
other partitions still have them, and the scan-all-partitions fallback
only engages after the home partition is exhausted, but it is
measurable, it is large, and it now reproduces on a second architecture.
The bound in 0006 addresses this, and to be clear I am not raising this
as an objection to partitioning. I mention it because if the bound is
going to be judged on whether it earns its place, this is the clearest
thing it does.

More benchmark details...

Pool occupancy was 81-89% rather than 100%, so there was a standing
supply of never-used buffers and the stock tails here are well below the
worst case I have measured at full occupancy, where stock's worst
allocation is exactly one full pass of the pool. The comparison between
arms is sound, since all three shared the same regime on each host, but
do not read the stock rows as an upper bound on stock.

The aarch64 box reports a single NUMA node. Its middle arm therefore
exercises the contention and eviction-predicate axes but not locality,
so that column is not a NUMA comparison. If someone has a multi-socket
arm64 machine I would be glad to see the locality half measured there.

Absolute throughput is not comparable across the two hosts, different
core counts and ~55% faster storage on the arm box, so only the
within-host deltas mean anything.

best.

-greg