Re: [PATCH] Batched clock sweep to reduce cross-socket atomic contention

On Wed, Jul 29, 2026, at 4:10 PM, Andres Freund wrote:
> Hi,
>
> On 2026-07-29 13:56:41 -0400, Greg Burd wrote:
>> AI NOTICE:
>> Yes, this patch was co-developed/refined/tested/benchmarked using
>> AI/LLMs and even this email started off as LLM content. All the work
>> is steered by me and reviewed before submission by me, I stand behind
>> it warts and all. (gulp) ;-P
>
> Not a fan. I'm ok with some closely supervised AI written code, but AI written
> emails I tend to just put to the bottom of my pile (which I rarely reach). I
> can't tell how much vetting goes into them, and I have too much to do to be an
> AI sanity checker.

Fair, and I'll stop using AI/LLMs in that way. I'd rather you judge the work than wonder how much I vetted it.

The last email was vetted by me. Was it too wordy - yes, was it a bit robot-voice - yes. I get it, human voice is "authentic" and that investment has meaning.

>> To find out whether the *mechanism* is real at all, I built a
>> sweep-bound microbenchmark: a function that pins+releases random blocks
>> of a table 1.7x shared_buffers with the normal (non-ring) strategy, so
>> StrategyGetBuffer is essentially the only work. 360 concurrent loops (=
>> cores on the instance), 64GB shared_buffers, 6 NUMA nodes:
>>
>> build block-evictions/sec vs stock
>> stock 2,096,314 --
>> bcs batch=1 2,138,485 +2% (single fetch-add: parity)
>> bcs batch=16 3,531,021 +68%
>> bcs batch=64 3,664,458 +75%
>
> Seems pretty doubtful this is measuring something super interesting, given
> that you'd a) use a lot of memory bandwidth for that much IO b) there's a lot
> of other contention points for that much IO (e.g. buffer table partition
> locks).

It is super-interesting because this condition does happen in practice with customer instances and has caused at least one cloud provider to create an internal work-around for it. Yes, there are other points where this workload has contention but right now I'm focused on this one.

> A 75% win actually seems somewhat disappointing for such a microbenchmark? I'd
> bet that you'd see WAY bigger gains with properly partitioning this (since the
> batching still leaves you with a lot of c2c accesses).

Maybe, but 75% isn't bad especially when the p99 during the overloaded state for clock-sweep is seconds to minutes and for HOT/COOL+batching is as steady as in any state. That's a real win IMO.

> I still don't see what the point of pursuing batching instead of helping out
> with Tomas' partitioned sweep patch is.

I'll do both, I appreciate the serious consideration of this algorithm. There's no reason the two ideas can't be combined. A partitioned HOT/COOL might be a solid improvement on its own, without partitioning.

>> 0002 -- cooling evictor: why the 0..5 counter, and the failure mode
>> ========================================================================
>>
>> Ants, you said the initial results were a promising indication that
>> G-CLOCK's 0..5 counter isn't carrying much value, and asked for a wider
>> access-pattern gamut. Here's the strongest thing I found.
>
> Bla.

Gesundheit.

>> The 0..5 counter has a pathological failure mode that the 1-bit HOT/COOL
>> clock does not. Stock must decrement a buffer from 5 to 0 before it can
>> evict it -- up to five visits per victim -- and it resets its
>> bounded-scan counter on every decrement, so under a hot,
>> continuously-evicting pool it never gives up early. With a large
>> shared_buffers and huge_pages off (a multi-GB BufferDescriptors array on
>> 4KB pages), those repeated visits are dTLB misses. I measured
>> ticks-per-victim climbing to ~9 on stock under load where the 1-bit
>> clock stays flat (it demotes HOT->COOL in one pass; a victim is produced
>> in at most ~3 ticks, and typically 1-2). In the extreme this turns into
>> a throughput cliff / multi-second p99 for stock -- the field reports of
>> "raise shared_buffers and TPS falls off" on large no-huge-pages
>> machines. The 1-bit clock has no such cliff because it does bounded
>> work per victim by construction.
>
> That's a lot of words with no actual evidence what the real world cost of that
> is.

This is the definition of the pathological case observed in production that Jimbo looked into when he was first trying out ideas that eventually led to the batching proposal.

The ticks-per-victim is a reasonable metric for "ability to make progress" and ~9 vs 3 can be meaningful in practice. The "real-world cost" is that your p99 latency spikes to seconds/minutes ore more in practice with the clock-sweep/0..5 algorithm but does not with HOT/COOL+batched.

>> On scan resistance, which is the reason the 0..5 counter's replacement
>> has to be careful: a demand-loaded page is admitted COOL (probationary)
>> and only promoted to HOT on a genuine second touch, so a one-touch
>> sequential scan fills and drains the COOL stage without displacing the
>> HOT working set. I tested this adversarially -- an OLTP hot set that
>> fits in shared_buffers, hit by point lookups, while a concurrent seqscan
>> streams a table 4x shared_buffers, on local NVMe (no EBS/IOPS cap
>> masking the eviction rate):
>>
>> OLTP hit ratio under the scan: stock 99.999% bcs 99.998%
>> OLTP resident set: flat in both
>
> I don't think it makes much sense to measure this when you can achieve close
> to 100 hit rate, since that's so easy to achieve. There's dozens of ways to
> achieve scan resistance that work well in that scenario but that fall entirely
> completely flat when you have actual cache pressure.

You're right that 99.99% hit rate is the easy regime and not the interesting one. The interesting question is testing your prediction, which I'll do. I'll see which page either algorithm protects/preserves in cache.

> I think this algorithm will *trivially* fail in a lot of cases without
> strategies. Once you have sufficient cache pressure the likelihood for
> cold->hot promotion ends up being too low to happen reliably before cold pages
> are evicted. And because this doesn't have something like a ghost directory,
> it'll not even have a chance to detect and fix that the next time the page is
> read in.
>
> I don't believe this algorithm will actually work well even without concurrent
> bulk access, I suspect it'll fail to work well even with something like
> pgbench -S where the index fits comfortably in s_b, but the table data does
> not. We already aren't good in that kind of workload, but I don't see how
> your algorithm will actually achieve *any* meaningful protection for anything
> but the inner index pages, the likelihood of repeat accesses to the same index
> page won't be high enough to keep those pages warm.
>
>
>> On the literature: you pointed at CLOCK-Pro, thanks -- the admission model
>> here (probationary COOL, promote on reuse) is the 2Q-A1 / test-period idea
>> and I'd rather adopt a better-justified promotion rule from that line than
>> defend "second touch" if the list prefers one. I tried a stricter "promote
>> only on a repeat reference" variant; it was neutral on OLTP and didn't
>> improve scan resistance, so I reverted it to keep 0002 minimal, but it's an
>> easy knob.
>
> FWIW, I really doubt that any of the scan resistant algorithms actually will
> allow us to get the read side strategies entirely, even with a ghost
> directory. All of the scan resistant algorithms only are scan resistant if
> repeated accesses (that should be cached) that occur during a concurrent bulk
> scan are actually within something like the size of the buffer pool, but that
> doesnt' really seem good enough.

Only one way to find out, I'll test this.

> That said, the current logic of strategies, where we actually evict the pages
> with a ringbuffer, seems rather wrong, because it takes approximately forever
> to fill the buffer pool even if there *are* repeat accesses.

Agreed.

> I suspect we should go for something like an extended version of
> S3-FIFO. S3-FIFO, has a FIFO for cold pages, taking up a certain percentage of
> the buffer pool (10% IIRC) and a ghost directory to detect repeat accesses
> that don't happen while the page is still in the cold fifo queue. That's scan
> resistant as long as repeat accesses are frequent enough (1.10x the buffer
> pool, I think).

That should be easy enough to test.

> To avoid bulk reads from pushing repeat accesses beyond the window in which
> repeat accesses are recognized, I think we could "just" make sure the ghost
> directory has eviction logic that will not allow strategy reads to take up
> more than a certain percentage of the ghost directory.

First, I appreciate your time reviewing this. I think it's time well spent and I'll try to make sure to voice my replies myself, warts and all.

Second, I'll review and add voice to Tomas's thread. This is a valid algorithm and I think they compose. I'll consider efficient ways to manage a ghost directory.

> Greetings,
>
> Andres Freund

best.

-greg