Re: [PATCH] Batched clock sweep to reduce cross-socket atomic contention

On Fri, Jul 10, 2026, at 2:07 PM, Greg Burd wrote:
> On Fri, Jul 10, 2026, at 10:00 AM, Greg Burd wrote:
>> Hello again,
>>
>> Removing the freelist [1] turned out to be a good idea, so why not remove
>> and simplify more?
>>
>> Included is a three-patch series (v3, on 2e6578292a9 cf-7000) that
>> replaces the 0..5 usage_count clock sweep with a cooling-stage clock,
>> and because this evictor algorithm is scan resistance 0003 removes
>> the BufferAccessStrategy ring buffers entirely. The net is slightly
>> faster (TPS), hotter buffers (fewer misses) and a lot less code to
>> maintain (no special case for scan resistant work) going forward.
>>
>> 0001 Batch the clock sweep to reduce nextVictimBuffer atomic contention
>> 0002 Replace the usage_count clock sweep with a cooling-stage evictor
>> 0003 Remove BufferAccessStrategy; scan resistance is now intrinsic
>>
>> Net it is +510/-1642 over 83 files; almost all of the deletion is 0003.
>>
>> On a 6-NUMA-node r8i.metal-96xl under continuous eviction, this change is
>> a consistent +4–5% on read-heavy pgbench with equal-or-better hit ratios.
>> The same test on a 2-node box is flat, which is the tell that the win
>> is the cross-socket clock-hand contention the batched sweep removes. Under
>> real storage IO (working set > RAM on local NVMe), the scan-resistance
>> path is +1.6–3.9% with measurably fewer heap reads.
>>
>> I'll put the design decisions and the benchmark methodology on the table
>> first, then talk through the results because if the model is wrong on
>> principle, or the benchmarks are measuring the wrong thing, I would much
>> rather hear it now.
>>
>> Contention arises on high core-count systems from the of cost advancing
>> nextVictimBuffer in StrategyGetBuffer() via a fetch-add of 1 per tick.
>> This contention is exacerbated on a multi-socket box. That one cache
>> line bounces over the interconnect on every eviction; under pressure
>> with hundreds of backends it is the dominant cost of the sweep. This is
>> why we're investing time on NUMA optimizations.
>>
>>
>> == What each patch does, and why ==
>>
>> 0001 -- Batch the clock sweep.
>>
>> This is the setup/baseline commit and is an incremental improvement
>> over the existing batched approach, but it is not the final goal and
>> should not be committed without 0002.
>>
>> Each backend will claim a run of consecutive buffer IDs with a single
>> fetch-add and iterate them privately. Global sweep order is preserved
>> -- each buffer is visited exactly once per pass -- only the intra-pass
>> ordering of visits changes, which the algorithm does not depend on.
>> The atomic fires ~1/batch as often.
>>
>> The batch is one cache line's worth of hand values (PG_CACHE_LINE_SIZE
>> / sizeof(uint32)), capped at NBuffers. It is gated on multi-node NUMA
>> (pg_numa_get_max_node() >= 1); on a single socket the batch is 1 and
>> the path is byte-identical to today. This is essentially Jim's patch;
>> the only substantive change from his v1/v2 is deriving the batch size
>> from the cache-line size rather than a fixed 64/tiered constant.
>>
>> 0002 -- Replace usage_count with a cooling-stage evictor.
>>
>> This is the heart of the series and where I most want the design
>> attacked.
>>
>> A buffer is HOT (recently used) or COOL (an eviction candidate);
>> "pinned" is the existing refcount. There is no per-buffer 0..5
>> counter. A demand-loaded page is admitted COOL (probationary); a
>> second access promotes it COOL->HOT. So a page touched once -- as is
>> the case with a sequential scan -- fills and drains the COOL stage and
>> is evicted from it without displacing the HOT working set. This is
>> the LeanStore / 2Q-A1 idea, and it is the whole reason 0003 can exist:
>> scan resistance stops being a ring-buffer bolt-on and becomes a
>> property of the replacement algorithm.
>>
>> The 4-bit usage_count field is reinterpreted in place -- bit 0
>> HOT/COOL, bit 1 a reference bit, the top 2 bits unused -- so the 64-bit
>> buffer-state layout, its refcount / flag / lock offsets. The only
>> change to the StaticAsserts is a new one asserting the field is at
>> least 2 bits wide, since we now use two of its bits. The eviction
>> claim (COOL, unpinned -> pinned) stays a CAS so a racing PinBuffer
>> always wins; promotion and demotion are single-bit transitions.
>>
>> The one non-obvious decision, which I got wrong first and want to flag
>> loudly: *Who* demotes HOT->COOL. My first cut was "prefer-COOL": the
>> foreground sweep skips HOT buffers hunting for an already-COOL victim
>> and only cools HOT buffers once a full pass finds none. That
>> collapses under an ordinary OLTP workload where the working set is
>> larger than shared_buffers (no scan at all). Every access promotes
>> its buffer to HOT, so there are almost no COOL buffers, the "cool a
>> full pass" fallback fires on nearly every allocation, and each victim
>> search becomes a ~2x full-pool scan of scattered BufferDescs. I
>> measured 3-17x throughput loss vs stock -- a cliff.
>>
>> HOT->COOL demotion is done during the background writer's existing LRU
>> scan, which already runs ahead of the clock hand. It demotes just
>> enough HOT buffers to keep a supply of COOL victims (bounded by the
>> predicted next-cycle allocation, so it does not cool the whole pool),
>> under the buffer header lock it already holds. A single reference bit
>> gives a recently-accessed buffer one reprieve before it is cooled,
>> which keeps the genuinely-hot set out of the COOL stage under scan
>> pressure. With that, the foreground finds a victim in a single pass
>> and the cliff is gone (data below).
>>
>> I chose prefer-COOL-plus-bgwriter-precooling because it can protect a
>> hot buffer slightly longer (the pre-cooler, tuned to a budget, decides
>> when to demote rather than demoting on contact), which should help hit
>> ratio on a stable hot set -- IF the pre-cooler keeps up. When it lags
>> (bgwriter off or behind), the foreground force_cool is still there as
>> a correctness fallback, but it is the expensive path.
>>
>> 0003 -- Remove BufferAccessStrategy.
>>
>> With scan resistance intrinsic, the BAS_BULKREAD/BULKWRITE/VACUUM
>> rings are dead weight, so this removes them end to end: the type and
>> enum, the ring machinery in freelist.c, the strategy parameter
>> threaded through ReadBufferExtended / the ExtendBufferedRel* family /
>> read_stream / every scan/vacuum/analyze/index-AM caller, the strategy
>> fields on the scan and bulk-insert descriptors, and
>> _hash_getbuf_with_strategy. pg_stat_io's per-strategy IO contexts
>> collapse to normal/init (IOOP_REUSE only ever happened while recycling
>> a ring buffer, so it is gone), and the vacuum_buffer_usage_limit GUC /
>> VACUUM (BUFFER_USAGE_LIMIT ...) option / vacuumdb --buffer-usage-limit
>> go with it.
>>
>> This is the patch most likely to be contentious for reasons unrelated
>> to the sweep: it removes a user-visible GUC and changes pg_stat_io's
>> shape. I have kept it as its own commit precisely so 0001+0002 can be
>> judged on the algorithm without swallowing the removal. If the
>> consensus is that the cooling model is fine but the ring machinery
>> should stay for other reasons (BUFFER_USAGE_LIMIT as an operator
>> control, say), 0003 can simply be dropped and 0002 still stands -- the
>> rings just become redundant rather than removed. I would like to know
>> if that is where people land.
>>
>>
>> == Benchmarks ==
>>
>> I want to be careful here, because Andres's central criticism of the
>> original batched-sweep numbers [2] was that they measured the wrong
>> thing -- an all-in-page-cache config where the only bottleneck left is
>> the atomic, which is not how anyone runs a 384-vCPU box. That criticism
>> is correct and I have tried to design around it, but I have almost
>> certainly not fully escaped it, so the methodology is laid out below in
>> enough detail to shoot at.
>>
>> I refer to the COOL/HOT approach (the changes in this patch set) as
>> "bcs", I've forgotten what that stands for... "buffer cache solution?"
>> I really don't remember (ha!).
>>
>> Hardware / method (both instances bare-metal, Amazon Linux 2023):
>>
>> - m6i.metal -- 128 vCPU, 2 sockets, 2 NUMA nodes (distances 10/20), 503GB
>> - r8i.metal-96xl -- 384 vCPU, 2 sockets, 6 NUMA nodes via SNC3, 3TB
>>
>> Builds: --buildtype=debugoptimized (-O2), cassert off, --with-libnuma,
>> both branches from the same tree. Postmaster pinned with numactl
>> --cpunodebind=0 --membind=0 (without it stock TPS varied ~30% by launch
>> node -- worth flagging for anyone reproducing). pgbench dataset loaded
>> once per build into its own datadir.
>>
>> The regime I settled on: a dataset that fits fully in OS page cache but
>> exceeds shared_buffers, warmed before each cell, caches NOT dropped
>> between cells. The point is a sweep that runs continuously with
>> sub-millisecond read latency and NO storage IO in the critical path --
>> so what I am measuring is the eviction machinery itself, not the disk
>> (as best as I can tell).
>>
>> I sweep the ratio (working-set / shared_buffers) from 0.8 (fits, no
>> eviction, a control) up to 8x by varying shared_buffers against a fixed
>> 63GB dataset.
>>
>> 3 iterations per cell, medians reported, 256 clients on m6i / 384 on r8i.
>>
>> Two workloads: uniform pgbench -S (the pure eviction-churn case, and the
>> worst case for the cooling model -- no hot set to protect), and
>> "hotscan" (a Zipfian hot set plus a handful of clients running large
>> range scans -- the scan-resistance case).
>>
>> I want the regime itself critiqued. It deliberately removes storage IO
>> to expose the sweep; the flip side is it is not a production
>> configuration, and the read-path win (fewer misses under scan
>> resistance) shows up as read count, not TPS, because a "miss" here is an
>> OS-cache memcpy, not a device read.
>>
>> r8i.metal-96xl, uniform pgbench -S, 384 clients, TPS (median of 3):
>>
>> ratio SB(GB) stock bcs delta stock/bcs hit%
>> 0.8 ~400 1,623,536 1,692,090 +4.2% 99.2 / 99.9
>> 1.25 ~320 1,443,343 1,519,290 +5.3% 94.8 / 95.2
>> 1.5 ~266 1,344,051 1,417,462 +5.5% 91.9 / 92.1
>> 2 ~200 1,293,871 1,358,550 +5.0% 87.9 / 87.8
>> 4 ~100 1,194,405 1,240,334 +3.8% 77.1 / 78.5
>> 8 ~50 1,090,873 1,140,839 +4.6% 69.7 / 70.6
>>
>> Consistent +4-5% across the eviction range, growing with pressure, with
>> equal-or-better hit ratio (better at 4x/8x). bcs also showed lower
>> cache-miss rate in perf stat (~28-35% vs ~31-37%), which is the batched
>> sweep's reduced cross-node line bouncing showing through.
>>
>> r8i, hotscan (Zipfian hot set + range scanners), TPS / hit% / heap reads:
>>
>> ratio stock TPS bcs TPS stock hit bcs hit stock reads bcs reads
>> 1.25 1,833,675 1,871,289 99.43 99.57 7,309,457 5,503,927
>> 1.5 1,876,846 1,942,877 99.35 99.42 8,335,144 7,565,667
>> 2 1,868,591 1,853,547 99.12 99.15 11,173,831 10,987,371
>>
>> The scan-resistance signal: at 1.25-1.5x, bcs holds a higher hit ratio
>> and does up to 25% fewer heap reads (24.7% at 1.25x, ~9% at 1.5x) -- it
>> is keeping the hot set resident through the scans where stock lets them
>> flush it. Muted in TPS only because everything is in OS cache (a miss
>> is cheap); on real storage this read reduction is where the win would
>> land. At 2x it washes out (enough pressure that both evict heavily).
>>
>> m6i.metal (2 nodes), uniform, 256 clients -- the smaller box, for contrast:
>> essentially parity, bcs -2% to +1% across ratios. The 2-node box barely
>> exercises the atomic, so 0001's contention win does not appear; that it does
>> not regress is the result that matters here.
>>
>> Huge pages on vs off (r8i, uniform, medians): I ran this because the
>> original thread flagged it as uncharacterized. bcs won by +3-7% both
>> ways, no regression without huge pages -- the win is from cutting the
>> frequency of atomic ops on the counter line, which does not depend on
>> where the descriptors physically live. (This is why the batching gate
>> is NUMA-only and not also huge-pages-gated.)
>>
>> The "prefer-COOL cliff" I mentioned under 0002, so the failure mode is
>> on the record: BEFORE moving cooling into the bgwriter, the m6i uniform
>> run at 256 clients was bcs 274K vs stock 840K at ratio 2 (-67%), and
>> 42.8K vs 762K at 8x (-94%), with cache-miss rate exploding to ~40%.
>> That is the shape of getting the demotion policy wrong; the r8i +5%
>> table above is after the fix.
>>
>> Reproduction: the whole harness (instance launch, OS tuning, per-build
>> load, the ratio sweep, perf stat capture) is scripted; I will attach it
>> as a DO-NOT-MERGE commit / put it in the CF entry so the methodology can
>> be reproduced and picked apart rather than taken on faith. Raw per-run
>> CSVs and perf output likewise.
>>
>>
>> A Real IO (working set > RAM, evictions hitting storage) Benchmark
>>
>> This is the regime Andres asked for, and the one I flagged earlier as not yet
>> done cleanly. The earlier attempt was EBS-latency-bound; this one uses local
>> NVMe so eviction reads hit real storage at ~microsecond, not ~15ms, latency --
>> during the run the array sat at 100% utilization and ~145K read IOPS, so the
>> eviction path is genuinely storage-bound, not cache-served.
>>
>> m6id.metal -- 128 vCPU, 2 sockets, 2 NUMA nodes, 503GB, 4x1.9TB local NVMe in
>> RAID0. Dataset ~700GB (pgbench scale 47000), i.e. LARGER than RAM, so the
>> working set cannot sit in the OS page cache. shared_buffers is a small window
>> over it -- 63GB (11x) and 31GB (22x) -- caches dropped per cell, 256 clients, 3
>> iterations, medians. Same builds/method as the in-cache runs otherwise.
>>
>> hotscan (Zipfian hot set + range scanners), median of 3:
>>
>> ratio SB(GB) stock TPS bcs TPS dTPS stock/bcs hit reads d
>> 11 63 877,357 911,363 +3.9% 96.30 / 96.43 -2.2%
>> 22 31 873,880 888,010 +1.6% 94.02 / 94.22 -1.5%
>>
>> This is the result the in-cache runs could only hint at: under real storage IO
>> the scan-resistance read reduction converts to throughput. The bcs approach
>> keeps a higher hit ratio and does 1.5-2.2% fewer heap reads, and here -- unlike
>> in cache, where a miss is a cheap memcpy -- a read it avoids is an NVMe round
>> trip, so the read reduction shows up as +1.6-3.9% TPS.
>>
>> uniform pgbench -S (pure eviction churn, no hot set to protect), median of 3:
>>
>> ratio SB(GB) stock TPS bcs TPS dTPS stock/bcs hit
>> 11 63 605,656 616,247 +1.7% 67.0 / 67.3
>> 22 31 605,517 611,312 +1.0% 63.5 / 63.6
>>
>> Hit ratio here is 63-67% -- a third of accesses miss and hit NVMe (190M-220M
>> evictions per run), so this is deep, genuinely storage-bound churn. bcs is
>> +1-1.7%, i.e. flat-to-slightly-positive, which is the honest production picture:
>> with no hot set to protect, scan resistance has nothing to do, and the win is
>> just the batched sweep's reduced contention showing faintly through the IO wait.
>> Notably bcs does not regress even when its policy has no advantage to exploit.
>>
>>
>> == Side note for the curious... ==
>>
>> Separately, Dhruv Aron has proposed restructuring the shared-buffer lookup table
>> [2], replacing dynahash with a flat two-array structure. That attacks the
>> other hot cost on a buffer miss — resolving a page to its buffer — where this
>> series attacks the eviction that a miss triggers. They touch buf_table.c and a
>> lock-ordering change in InvalidateBuffer(); this series touches freelist.c and
>> the per-buffer replacement state, and removes BufferAccessStrategy. The two are
>> complementary and should compound on the miss path; the only overlap is
>> InvalidateBuffer()/GetVictimBuffer(), where their extended buffer-header-lock
>> hold and this series' CAS-claim + bgwriter pre-cooling both take that lock, and
>> would want reconciling if both land. I have not benchmarked them together (yet).
>>
>> == What I have not done, honestly ==
>>
>> - Hardening the foreground force_cool fallback to be cheap when it
>> fires, rather than relying on the bgwriter pre-cooler never lagging.
>>
>> - Anything on single-socket beyond "does not regress"; the design is
>> not trying to help there.
>>
>>
>> == The ask ==
>>
>> 1. 0002's demotion policy: is prefer-COOL + bgwriter pre-cooling the
>> right call, or is the other team's cool-in-place the more robust
>> default given it has no cliff and no background-process dependency?
>> This is the decision everything else hangs on.
>>
>> 2. Is admitting demand-loaded pages COOL (probationary,
>> promote-on-second- touch) an acceptable basis for scan resistance
>> in the core buffer manager, i.e. is it OK to make scan resistance
>> an algorithm property and retire the strategy rings (0003)? Or
>> should the rings stay and 0002 ride alongside them?
>>
>> 3. The benchmark methodology: where is the in-OS-cache regime
>> misleading, and what would you want measured instead? I am most
>> worried I am flattering the sweep by removing the IO that would
>> otherwise hide it.
>>
>> 4. Reinterpreting the usage_count field as {HOT/COOL, ref} bits and
>> collapsing pg_stat_io's contexts -- acceptable, or is there a
>> cleaner representation the project would want before this is worth
>> pursuing?
>>
>> I have measured that the 0..5 count is overhead and provides no
>> meaningful signal at all, that a HOT/COLD approach provides a simpler
>> more stable and better performing eviction model for the buffer pool.
>> If you dispute that, let's dig in and compare notes. :)
>>
>> I would be remiss if I didn't point out the thread [3] by Tomas et. al.,
>> whose NUMA investigation targets the same bottlenecks, and inspired the
>> work that led to this set of ideas.
>>
>> Thanks for reading this far. I look forward to the critique.
>>
>> best.
>>
>> -greg
>>
>> [1] Reconsidering the freelist
>>
>> https://www.postgresql.org/message-id/f0e3c02e-e217-4f04-8dab-1e7e80a228c0%40burd.me
>> [2] Re: Restructured Shared Buffer Hash Table
>>
>> https://www.postgresql.org/message-id/dbbd1998-19ff-4ac2-b4b1-a39f4ec1b0f5@iki.fi
>> [3] Adding basic NUMA awareness (Tomas Vondra)
>>
>> https://www.postgresql.org/message-id/099b9433-2855-4f1b-b421-d078a5d82017%40vondra.me
>> Attachments:
>> * v3-0001-Batch-the-clock-sweep-to-reduce-nextVictimBuffer-.patch
>> * v3-0002-Replace-the-usage_count-clock-sweep-with-a-coolin.patch
>> * v3-0003-Remove-BufferAccessStrategy-scan-resistance-is-no.patch
>
> Rebased v4 onto c71d43025d7.
>
> -greg
> Attachments:
> * v4-0001-Batch-the-clock-sweep-to-reduce-nextVictimBuffer-.patch
> * v4-0002-Replace-the-usage_count-clock-sweep-with-a-coolin.patch
> * v4-0003-Remove-BufferAccessStrategy-scan-resistance-is-no.patch

Rebased v5 onto 5f14f82280d to keep the CF-bot happy (and everyone else, I'd imagine).

-greg