Re: [PATCH] Batched clock sweep to reduce cross-socket atomic contention

Hello again.

Well, I've been using the past few days to dig into the
HOT/COOL+batching model and I think I've found the corners and have
data to support it. Attached find v8 (on 9fa2c1e) and a tar.gz with
my benchmark.

TL;DR:
Empirically-grounded data shows that HOT/COOL+batching:
- (a) provides scan resistance which is net new outside of the
ring buffer
- (b) is simpler (1-bit vs 0..5) <- that's a lot fewer bits, lol!
- (c) I've found no evidence of any signal provided by the 0..5
clock-sweep counter and direct evidence that it can, in
limited but not-unimaginable cases, become the root of
pathologically bad performance
- (d) batching de-contends nextVictimBuffer on NUMA +68-75% when
the sweep is hot, no regression otherwise. Not "always
faster," but "targeted de-contention that pays off exactly
when the sweep saturates, and never hurts when pooled.

I've tested with pgbench, HammerDB (TPROC-C/H), and custom code to
model and elicit the worst/best cases and the pay-off cases I could
imagine. I've not done this on local (non-cloud) hardware (I'm
building that out now!) but on EC2 metal instances which I'll argue
are close enough and all I had available (for now).

I may still be benchmarking in ways that you feel don't elicit the
best, most accurate results. Feel free to point that out and I'll
adapt and re-run the tests.

AI NOTICE:
Yes, this patch was co-developed/refined/tested/benchmarked using
AI/LLMs and even this email started off as LLM content. All the work
is steered by me and reviewed before submission by me, I stand behind
it warts and all. (gulp) ;-P

DETAILS:
I've done quite a bit of measurement. This reply is long because
it needs to be, legitimate requests for real data required thought
and time to execute, analyze, etc. Short version of what changed and
what I found:

- Two patches now, not three. I dropped the BufferAccessStrategy
removal; it removed real bulk-writer backpressure with no
replacement (Ants and Andres were both right), so it's out until
that's solved separately.
- I withdrew the v6/7 dirty-bulk "regression" and VACUUM "win" as
both were mismeasurements, detailed in my earlier replies.
- The batched sweep (0001) and the cooling evictor (0002) are
independent and I'm treating them as such; 0001 stands alone if 0002
is contentious, but used together is the real win.
- I built a sweep-bound microbenchmark that isolates the
nextVictimBuffer atomic, and it settles the batching question with
perf c2c evidence (below). It also shows honestly where batching
does *not* move the needle.

The series:

0001 Batch the clock sweep to reduce nextVictimBuffer atomic contention
(Jim Mlodgenski's idea; co-authored)
0002 Replace the 0..5 usage_count clock sweep with a 1-bit HOT/COOL
cooling-stage evictor (LeanStore/2Q-A1), rings kept

All numbers below are on AWS bare metal, working set sized to fit RAM so
misses are OS-page-cache-cheap (this is a CPU/TLB study, not a storage
one -- I'll say explicitly where that matters). Instrumentation (a
throwaway ticks-per-victim counter and a block-thrash function) is not
part of the patches; the repro kit is attached as repro.tgz_ (underscore
to avoid being picked up by CI I hope).

0001 -- batched sweep: does it help NUMA, and does it regress?
========================================================================

Andres, in the first reply on this thread you doubted that batching
independent of contention/usage-rate would help, and suspected it might
hurt. I set out to prove or disprove that, and the answer is: it depends
entirely on whether the clock sweep is the bottleneck, and I can now
show exactly when that is the case.

First, the fair-throughput question. Earlier I got confusing results
because I was running 2000-8000 backends against a 384-vCPU box -- 5-20x
oversubscribed -- which is a scheduler-storm test, not a batching test,
and there batching *did* look like it hurt (~-35%). With a connection
count at or below core count (the pooled configuration anyone runs in
production), on a 6-NUMA r8i.metal (384 vCPU), uniform pgbench-style
point lookups, stock vs batch{1,16,64} are all within ~1-2% at every
level -- batching neither helps nor hurts. So: no regression when
you're not oversubscribed; the -35% was an artifact of oversubscription,
which a connection pooler removes.

Why neutral there? Because in a normal point-lookup the executor +
protocol + tuple deform dwarf the one atomic; even stock only did ~2.9
clock ticks per victim, so nextVictimBuffer just wasn't hot enough to
matter.

To find out whether the *mechanism* is real at all, I built a
sweep-bound microbenchmark: a function that pins+releases random blocks
of a table 1.7x shared_buffers with the normal (non-ring) strategy, so
StrategyGetBuffer is essentially the only work. 360 concurrent loops (=
cores on the instance), 64GB shared_buffers, 6 NUMA nodes:

build block-evictions/sec vs stock
stock 2,096,314 --
bcs batch=1 2,138,485 +2% (single fetch-add: parity)
bcs batch=16 3,531,021 +68%
bcs batch=64 3,664,458 +75%

and the mechanism, straight from perf c2c (cross-node HITM on the
hottest line):

stock: StrategyGetBuffer's fetch-add on nextVictimBuffer is the #1
cross-node HITM line at 38.5%, spread across all 6 nodes.
batch=16/64: that same line drops to ~1-2% HITM.

So the batched claim does exactly what it claims: nextVictimBuffer is
touched ~1/N as often, its cross-socket bounce collapses, and when the
sweep is the bottleneck throughput rises 68-75%. batch=1 == stock
confirms the win is purely the batched atomic, not the evictor.

The honest framing, which I'd rather state than have pulled out of me:
this is a microbenchmark that isolates the atomic. It proves the
mechanism is real and bounds the ceiling; it does not claim a 68% win on
your OLTP box. On real mixed workloads at backends<=cores the sweep is
a small share of each query, so end-to-end it's neutral -- no
regression, no headline speedup -- until the sweep actually saturates
(large hot pool, heavy eviction, many NUMA nodes, which is the reported
field pathology). 0001 is a targeted de-contention that pays off
exactly when the hand is hot and costs nothing when it isn't; on a
single socket it compiles to the stock path (batch size 1).

I did also run HammerDB TPROC-C at backends<=cores; it's
write/lock/WAL-bound, the sweep is not the bottleneck, and batch=64 was
~5% *below* stock (the cooling bookkeeping isn't free when the sweep
isn't the cost). Reporting that too -- it's not a batching workload,
and I'm not going to pretend it is.

0002 -- cooling evictor: why the 0..5 counter, and the failure mode
========================================================================

Ants, you said the initial results were a promising indication that
G-CLOCK's 0..5 counter isn't carrying much value, and asked for a wider
access-pattern gamut. Here's the strongest thing I found.

The 0..5 counter has a pathological failure mode that the 1-bit HOT/COOL
clock does not. Stock must decrement a buffer from 5 to 0 before it can
evict it -- up to five visits per victim -- and it resets its
bounded-scan counter on every decrement, so under a hot,
continuously-evicting pool it never gives up early. With a large
shared_buffers and huge_pages off (a multi-GB BufferDescriptors array on
4KB pages), those repeated visits are dTLB misses. I measured
ticks-per-victim climbing to ~9 on stock under load where the 1-bit
clock stays flat (it demotes HOT->COOL in one pass; a victim is produced
in at most ~3 ticks, and typically 1-2). In the extreme this turns into
a throughput cliff / multi-second p99 for stock -- the field reports of
"raise shared_buffers and TPS falls off" on large no-huge-pages
machines. The 1-bit clock has no such cliff because it does bounded
work per victim by construction.

On scan resistance, which is the reason the 0..5 counter's replacement
has to be careful: a demand-loaded page is admitted COOL (probationary)
and only promoted to HOT on a genuine second touch, so a one-touch
sequential scan fills and drains the COOL stage without displacing the
HOT working set. I tested this adversarially -- an OLTP hot set that
fits in shared_buffers, hit by point lookups, while a concurrent seqscan
streams a table 4x shared_buffers, on local NVMe (no EBS/IOPS cap
masking the eviction rate):

OLTP hit ratio under the scan: stock 99.999% bcs 99.998%
OLTP resident set: flat in both

i.e. the cooling evictor's scan resistance is at parity with what the
BAS_BULKREAD ring gives today. (An earlier version of this patch also
removed that read ring on the theory that COOL admission made it
redundant; local-NVMe testing showed it wasn't quite -- the ring is
measurably tighter under concurrent scan -- so I kept the ring. That's
the dropped 0003.)

On the literature: you pointed at CLOCK-Pro, thanks -- the admission
model here (probationary COOL, promote on reuse) is the 2Q-A1 /
test-period idea and I'd rather adopt a better-justified promotion rule
from that line than defend "second touch" if the list prefers one. I
tried a stricter "promote only on a repeat reference" variant; it was
neutral on OLTP and didn't improve scan resistance, so I reverted it to
keep 0002 minimal, but it's an easy knob.

On backpressure: I'm not removing the rings, precisely because
BAS_BULKWRITE is backpressure, not just a pollution guard. If we ever
do want the rings gone, that backpressure needs a real replacement
first; I don't have one worth proposing, so it's out of scope here.

Relationship to the NUMA partitioned sweep
========================================================================

Tomas, this overlaps your NUMA series and I want to be clear it's not a
competitor. 0001 attacks the same nextVictimBuffer cross-node
contention your partitioned sweep does, but as a minimal,
placement-agnostic change: one fetch-add per batch instead of per tick,
no per-node partitioning, no buffer-placement changes, batch-size-1
fallback so it's a no-op off NUMA. The c2c data above is the same
contention your partitioning targets, measured in isolation.

I think these compose rather than conflict: a partitioned sweep still
advances a hand within each partition, and batching that per-partition
hand should give the same cross-node relief inside a partition that it
gives the global hand here. If your series lands, 0001 is a small
change on top (or moot, if the partition hands are already node-local
and uncontended). If it's useful I'm happy to test batching layered on
your patches -- I just didn't want to fold placement/partitioning into
this, since the evictor and the atomic de-contention stand on their own
and are far smaller.

What I'm claiming, and what I'm not
========================================================================

- 0002 is simpler (1 bit vs a 0..5 counter), has intrinsic scan
resistance at parity with the BAS_BULKREAD ring, and removes a real
large-pool/no-huge-pages cliff that the 0..5 counter can hit. It is
throughput-neutral on ordinary OLTP (within noise), which is the bar
for a replacement-policy change.
- 0001 de-contends nextVictimBuffer on NUMA: +68-75% when the sweep is
the bottleneck (c2c-proven mechanism), neutral otherwise, no-op off
NUMA, no regression when connections are kept at/under core count.
One idea would be to lower batch size to 1 when the number of
backends > cores available on the host to avoid the regression, but
that isn't in v8 nor have I tested it in practice.
- I am NOT claiming a general TPS win; on write/lock-bound or
non-sweep-bound workloads there isn't one, and I've shown that.

Repro kit attached (methodology, the microbenchmark, the ticks/victim
and block-thrash instruments, the residency-fraction model that predicts
when the 0..5 cliff appears, and the huge_pages on/off and
confound-discrimination notes). It's built to be re-run and attacked;
if any of this doesn't reproduce for you, that's the feedback I want.

Thanks for the reviews, they made the patch set smaller and the claims
honest.

best.

-greg