Re: Adding basic NUMA awareness

I need to correct the numbers I posted on the 15th. I rebuilt the comparison
after finding problems in my own harness, and the effect is real but about ten
times smaller than I reported.

I corrected the mistakes and re-ran both arms from the same base differing only
in the evictor, no batching in either, counters snapshotted after warmup and
subtracted, 5 runs alternated, s_b=2GB against an 8GB heap with three 850MB
secondary indexes, zipfian index probes. The non-NUMA numbers are from a
single-socket machine this time, and the postmaster is pinned on the NUMA box.

stock 0..5 HOT/COOL delta
single socket (1 node, 64 clients):
index-leaf resident 106,721 107,929 +1.13%
index-inner resident 1,164 1,164 same
index share of evictions 36.89% 36.81% -0.08pt
index hit ratio 98.6473% 98.6541% +0.007pt
tps 242,104 241,253 -0.35%
2 nodes, pinned (192 clients):
index-leaf resident 55,374 55,781 +0.74%
index-inner resident 1,164 1,164 same
index share of evictions 40.28% 40.21% -0.08pt
tps 295,393 299,150 +1.27%

So, with hot/cool the direction is consistent and the per-run ranges for the
residency and eviction-share columns don't overlap, but the magnitude is ~1%
more index leaf pages held and 0.08 points less index eviction, not the 0.8
points I claimed.

Index-inner residency is identical in every condition, both hot/cool and
clocksweep0..5 keep the whole inner set, so my "protects frequently-used pages"
framing was too broad; it's a leaf-page effect. Hit ratio moves by thousandths
of a point and TPS is inside the run spread either way.

One methodology note in case it saves someone else time: unpinned, the
2-node TPS difference came out at -7.5%, and pinning the postmaster
turned it into +1.27%. Placement noise was larger than the thing I was
trying to measure.

Attached patches are rebased on current master and replace the v20260915
set. Changes to v20260922a beyond the rebase:

- The reference bit is gone. It wasn't needed: the sweep demotes
HOT->COOL on the visit and only reclaims on a later tick, so a
buffer already gets a full sweep of grace to be re-promoted. The
replacement state is one bit now.
- A strategy access no longer promotes, which restores the existing
rule that ring buffers don't evict others from the pool and makes
GetBufferFromRing()'s reuse test correct.
- Dropped the bgwriter changes I had bundled in (smoothing window and
the write cap). 0009 is a replacement-policy change only now.
- Removed a dead pre-cooling path and a dead trycounter branch, and
rewrote the commit message, which still described batching and ring
removal that aren't in this series.

At this point I think that while hot/cool works and IMO is simpler it's not
worth the potential risks and isn't justified in terms of performance which
should be the main driver for a change like this. I'll drop it and just
continue with the clocksweep and NUMA changes from here on unless someone
else is supportive.

best.

-greg