Re: Adding basic NUMA awareness

On Thu, Jul 2, 2026 at 11:24 AM Jakub Wartak
<jakub(dot)wartak(at)enterprisedb(dot)com> wrote:
>
> On Tue, Jun 30, 2026 at 2:51 PM Jakub Wartak
> <jakub(dot)wartak(at)enterprisedb(dot)com> wrote:
> >
> > On Mon, Jun 29, 2026 at 9:42 AM Jakub Wartak
> > <jakub(dot)wartak(at)enterprisedb(dot)com> wrote:
> > >
> > > On Thu, Jun 25, 2026 at 3:49 PM Tomas Vondra <tomas(at)vondra(dot)me> wrote:
> > > >
> > > > >> I have some results from a new round of benchmarks, and it's a bit
> > > > >> disappointing. Or rather, there seem to be some issues that I can't
> > > > >> figure out, causing regressions.
> > > > > [..]
> > > > >> This chart is for median latency (in milliseconds):
> > > > >>
> > > > >> clients master 0003 0004 0003/on 0004/on
> > > > >> -------------------------------------------------------------
> > > > >> 1 12767 12582 14509 12807 15307
> > > > >> 8 14383 14355 14149 14069 16165
> > > > >> 32 14756 15198 14836 14984 17128
> > > > >> --------------------------------------------------------
> > > > >> 1 103% 114% 100% 120%
> > > > >> 8 101% 98% 98% 112%
> > > > >> 32 102% 101% 102% 116%
> > > > >>
> >
> > [..lots of variables..]
> >
> > > > I'll try, but if you could try running some experiments on your own,
> > > > that might be helpful.
> > > [..]
> > > > > Hopefully next week I'll try to repro those numbers to see if I can
> > > > > help more.
> > > > >
> > > >
> > > > Thank you! That'd be great.
> > >
> > > Yeah, I'll try my best, we'll see how it goes. Right now I've just dropped
> > > that fscachenuma proggie to aid us in troubleshooting.
> > >
> > > -J.
> > >
> > > [0] - https://github.com/jakubwartakEDB/fscachenuma
> >
> > Hi Tomas,
> >
> > OK, so I've run couple of tests and modified run.sh and also tried to fix
> > some inefficiencies spotted while testing this. Note the attached
> > performance matrix is in TPS (so more is better). Raw results/CSV and
> > scripts are attached too.
> >
> > * run2 = 2 workloads, partitioned pgbench_accounts
> > * run3 = just pgbenchS w/o partitioning + warmup
> > * run4 = semi-like pgbenchS w/o partitioning but 100k rows + warmup
> >
> [..]
> >
> > Stil on the table:
> >
> > 1. maybe even better balancing is possible (?), but this one is seems enough?
> > I'm out of other ideas, well other than the
> > "shared-relation-use-by-foreign-node" idea described much earlier (but
> > I won't be able to pull that off), so I'm not entering this rabbit hole
> > any deeper.
>
> See below, seems like not needed (?)
>
> > 2. Digging into io_method=worker optimizations (answering question: are they
> > necessary?) Maybe I'll throw in run5 quite soon, this is going to be
> > crucial to answer.
>
> OK, I'm attaching are results from mine runs 5 and 6:
> - only seqconcurrscans was tested, well because for other workloads io_worker
> method was not getting load for those workers (only seq scans were offloaded)
> - checksums were disabled, because IMHO that would be unfair comparision
> (AFAIR there are offloaded)
> - those optimizations for 0008 "optimized (numa=on, bal=on)" easily beat
> "patched (numa=on, bal=on)" and seem to be crucial. We get like 1.2x-1.4x
> across every io_method, but only with 0008.
> - even when then doing just those logical fully cached reads from fully VFS
> cached case, io_urings shines (I've added raw TPS number to show this,
> compare across tables e.g. io_uring vs sync 13.378/8.993=1.487x for
> io_uring with NUMA, but for master's for io_uring:sync it was just 8.79/7.389
> = 1.189x without NUMA); seems like io_uring is more lightweight to show
> more benefits of remote memory latencies
> - there's some more juice to get out of the balancer for 0-reuse workloads
> (but IMHO it's pointless to squeeze more, it's hard already)
> - I was probably wrong when expecting that io_worker's worker processes/queues
> should get NUMA affinity. They don't need to be apparently for me to see
> benefits (maybe they could be and it would even better, but meh).
> So with ruling io_method impact (I speculated earlier that his could be it),
> this means that you were either hitting lack of opimizations needed from
> 0008 or were impacted by lack of drop_caches before the runs
>
> > Maybe You could run those run*.sh with master vs inst-patchset/optimized?
> > (I'm not sure, maybe there's even different factor at play too...)
>
> This is seems to be crucial now, to double confirm the results / loaded-tested
> on your hw with 0008. (but that hardware really needs to have effective latency
> difference between at least 2 NUMA nodes -- Intel's mlc is good for this);
> maybe also tweak those 125% inside 0008 to some other values, I've got 4 nodes,
> so 100/4=25%)
>
> > 3. Potentially mentioned earlier BAS strategies (forcing just use of local
> > partitions for known-to-be-only-local-users: CTAS/VACCUM/etc), but I'm
> > afarid that's not for me as I would certainly break/violate some
> > invisible to me boundary.
>
> And this one is still potentially on the table as nice thing to have.

Hi Tomas,

I've took another look at this thread and got some new further progress:

TL;DR;
1. We seem to crash on CXL setups and need some auto-detection of system
(code).
2. Verification of patchset + some further tweaks gives ~120% for
seqconcurrscans on modern hardware.
3. Doubts why we are not getting more, how many partitions , HB176v4 and so
on.
3a. 1 client results explanation theory
3b. Benchmarking $this optimized patchset against Greg's/Jim's batched to see
how much potential juice we have left on of the table (=> not much)
4. Some self-relfections on my previous BAS strategy idea (== meh!)

---

0. Patch needs rebase. For below I've just used master as of Jul 2026

1. In order to verify if we even reach the lowest bar (not crash): Andres
wrote:
> Given the relative unavailability of CXL memory systems, I think just
> not crashing is good enough for now[..]

I've found nice/fast way to verify CXL without access to CXL hardware but
w/o CXL latency simulation. With this CPU-less NUMA setup which boils down
to commands that setup QEmu, we do crash:

qemu-img create -f qcow2 debian.qemu 10G
qemu-system-x86_64 -smp 4 -m 8G -enable-kvm \
-object memory-backend-ram,id=ram0,size=4G \
-object memory-backend-ram,id=ram1,size=4G \
-numa node,nodeid=0,cpus=0-3,memdev=ram0 \
-numa node,nodeid=1,memdev=ram1 \
-numa dist,src=0,dst=1,val=30 \
-hda debian.qemu

In the above nodeid=1, has no cpus and that results in CXL-like thingy as
per numactl --hardware/libnuma goes. Of course one needs to install using ISO
on the emulator and try to apply the $patchset. So if I add hugepages
just to node0 (via /sys/devices/system/node/) then we get failure to start
with "Bus error" with failure coming from ShmemInitRequested()->
BufferManagerShmemInit()->ConditionVariableInit(). Anyway, the code today
just did blind mbind() to that CPU-less NUMA-node (e.g. node1) where we do
not have hugepages , so it crashes with SIGBUS.

After adding hugepages to node1 too (CPU-less one), of course it boots
(similiar to sysctl's -w combining nr_hugepages setup which is divided into
multiple nodes), but then it will start using memory from that simulated
node1 CXL memory - which would be terrible for latency (degraded system?).

So within the patchset in 0003 in BufferPartitionsCalcuate() assumes
nnodes = numa_num_configured_nodes();
nparts_per_node = 1; /* at least one partition per node */
but those two lines are wrong. We have two options:
a) detect CPU-less nodes and avoid mbind()ing to them (assume they are
CXL)
b) or have explict configuration to which NUMA node we want allocate memory

AFAIR my understanding is that You wanted to avoid configuraiton as long
as possible and just stick to what's available/exposed by the OS that could
be usable, so I've tried wit attached n1 [0] proggie to see if that's
possible (to certain extent it is, check n1*txt files and especially "way3"
to result how we could detect such setups and avoid mbind() and SIGBUS).

Orthogonal thing for configuration is CPU where we can run on (
sched_getaffinity() and similiar). There's also need for handling another
situation: some users could use numactl/systemd to force NUMAMask (check
systemd.exec(5)) and we should auto-detect that and not issue mbind()
there too.

So, the bigger problem with all of that is that the NUMA nodes cannot be
consecutive loops as it is (for PoC it's fine, but especially with CXL you
can get hotplugged stuff that appears / disappears and so on, so I think we
cannot assume that simple loop to mbind(n):
for nodes = 0..#numa_num_configured_nodes_nodes() is fine, it seems it
isn't.

2. You guys doubted my hardware, so I've got my hands on much more modern
i4i.metal on AWS (with specs like 2s64c128t and 2 NUMA real nodes with
105:170ns latency difference according to Intel's mlc and 175GB/s vs 55GB/s
bw for local:remote QPI speeds).

run7/20260825 shows it that "optimized" (so thats +0008 patch) clearly wins
against not having it across all three io_method modes on that modern
hardware too. In essence the patchset (w/ 0008) couldn't give pgbenchS boost,
but gives ~9-10% in seqconcurrscans. I was kind of woried about those
1-1.5% pgbenchS/pgbenchS100krows regressions for optimized though, so I've
started looking for more and took a look on number of clocksweep partitions
and putting stuff into dedicated cachelines (especially nextVictimBuffer
even when it would be partitioned, because why not to try it blind):

run12 20260828 cover comparision between raw checking impact of number of
partitions vs
- master (from around Jul 2026)
- optimized-aligned (+0008, +pg_attribute_aligned(PG_CACHE_LINE_SIZE),
- optimized-16parts (MIN_BUFFER_PARTITIONS=16)
- optimized-aligned-16parts (+0008, +pg_attribute_aligned, 16 parts too),

It seems that forcing patchset certain struct members does provide minimal
boost, (1-2%? for bal=on), but if anything then 16 clock-sweep partitions
help a little bit more benefit and get us into 120%+ for seqconcurrscans
without regressions from pgbenchS.

However this is started kind of worrying to me.

3. This is kind of worrying and sad of not getting better results (1 year there
was this quote by Tomas):

> That's not bad - the clocksweep partitioning increases the throughput
> 2-3x. Having 16 partitions (instead of 4) helps yet a bit more, to 3-4x.

So i've got the following results for patchset+0008+aligned still on that
i4i.metal when measuring with perf that seqconcurrscans -c 128:

clocksweep partitions | StrategyGetBuffer() CPU% | TPS | --vs-baseline
1 (master, baseline) | 9.09% | 68 | -
2 (1 per node,optimized) | 4.84% | 73 | 107.3%
16 (8 per node,optimized)| 0.32% | 79 | 116.3%

Certainly far from 2-3-4x reported earlier. And then:

> For epyc (hb176), the differences are pretty massive.

I dont have acess to hb176v4, to replicate those results (maybe it's
special, EPYC with 4 nodes?? and apparently BIOS set to NPS=2, L3-as-NUMA
disabled), but Your's testing generate.sh from that old incarnation of
patchset had some stuff like that:
CREATE TABLE x (id int, val text) WITH (fillfactor=10);
INSERT INTO x SELECT 1, md5(i::text) FROM generate_series(1, 1442000);
Which is exactly 1GB (!), and single such socket Genona-X handles 1152MB of
3D V-Cache alone, so sounds like it? (somehow unlocking CPU prefetching
everything from DRAM to such enormous L3 in one go? maybe something with
lower dTLB misses because of that)

Anyway, I've started digging this time on my legacy 4s32c64t 4 nodes box
(with just 4x 64 MB LLC :P) to find max optimal seqconcurrscans
partitions vs
clients and got this TPS results for similiar seqconcurrscans:

clients | master | ppn=1 (4 parts) | ppn=2 (8 parts) | ppn=4 (16 parts)
1 | 0.714 | 0.457 (−25%) | 0.361 (−47%) | 0.351 (−49%)
8 | 2.177 | 2.567 (+19%) | 2.515 (+21%) | 2.577 (+21%)
32 | 5.664 | 7.238 (+29%) | 7.722 (+37%) | 7.887 (+39%)
64 | 9.201 | 11.675 (+31%) | 12.486 (+37%) | 12.646 (+38%)
(ppn=partitions per node, that is 4s32c64 4 NUMA nodes box, that was quick
look without turbo boost disabled and so on)

3a.mini-problem A): IMHO the c=1 regressions are worrying and I think I saw
them seen earlier, so it's not that we can just throw more parts, right ?
It looks it's just clocksweep part. size (NBuffers / parts) vs. the
scanned relation size (1.6GB here, under NBuffers/4 => does not use ring
buffer for seq scans). So with s_b=8GB the per-partiton size is:
- 2GB (with 4 parts)
- 1GB (with 8 parts) and so on
- against full 8 GB which is undivided on master.
At c=1 only one table is touched all the time, so on master the whole
table becomes resident and all scans just hit s_b. Once a backend is
locked to a partition smaller than 1.6 GB (8 parts) it can never retain
the table and stuff degrades (spreading starts to cost us too much?). So
that would indicate 1-2 partitions per node is OKayish, but more is not?
(but that's contrary to what Andres I think mentioned during live
presentation to have partitioned clocksweep even per CPU?)

3b.bigger doubt B): why we are not getting more juice here if in the past
people did report more? Even worse Greg & Jim, reported in [1] with
simplest form of batch-updating the nextVictimBuffer consistent reports
up to +16..20% for __pgbenchS(RO)__, but here we cannot even do so for
seqconcurrscans?

So I've compared those two ideas against each other, result in run15:
- master
- inst-optimized-aligned was still with just 4 parts (on that 4s32c64/4
nodes) and with that pg_attribute_aligned thingy
- batchedincr was same master, but just with Jim's/Greg v1 addon.

I couldn't see any increase of pgbenchS (RO), nor even 2x for
seqconcurrscans, but I could see that this patchset more or less gives
1.2x today, while their batched-atomic-fetches yield ~1.3x.

Anyway, I think that crosscheck result is quite good result, because this
thread/patchset+some optimizations is right path a good basis for building
further optimizations in the future.

4. My earlier BAS strategy idea == meh.
BAS strategy (BufferAccessStrategy-derived affinity; local-only rings for
certain operations): I've tried to implement that idea from [2], but it
appears that even if I forced ring assigned from local NUMA node memory
(e.g. for CTAS from generate_series) it realy doesn't matter, as
everything is rarely touching remote DRAM, most of stuff is simply cached
by L1 (multiple hits into same *page* < L1d cache size) or L2/L3 and that
hides the remote latency.

The only thing that actually could benefit is the random btree index probe
hitting remote latencies (that would be hitting more than L3/LLC size, but
still within s_b size), stuff that would disarm CPU memory prefetcher.

However with +0008 allocator already seems to be placing lots of stuff on
local node first if possible, so it appears we seem not need anything
else to benefit from special BAS that would pin it to just local s_b in
places where we know about. I think that's good sign actually (less code).

BTW: I've failed to even come-up with operation that would:
1. be dependent on latency , random memory fetching that would stall (low
IPC, random, not prefetchable)
2. The data being larger than L3/LLC, to ensure we hit remote DRAM really
3. Remote DRAM data that would be repeatably hammered (with #2 being
larger than L3/LLC)
4. Not being blocked by I/O

Maybe something with HashJoin, dunno, way out of my area.

4b.The only thing that showed some promise would be meta-data (such as
SharedBufHash?), but realistically with those NUMA latencies effects, I
could just get like 3-4% improvement if force-pinning it to loca/remote
memories just to see what would be the effect..

-J.

[0] - https://github.com/jakubwartakEDB/n1

[1] - https://www.postgresql.org/message-id/79629577-3ad8-4b1c-a469-ebc2cb4c5104%40app.fastmail.com:
r8i.metal-96xl (384 vCPUs, 2 sockets, 6 NUMA nodes via SNC3):
pgbench RO:
Clients Stock Patched Delta
64 31,457 36,353 +16%
128 31,678 37,864 +20%
[..]

[2] - "I. BufferAccessStrategy: we could derrive affinity from the BAS
strategy itself, couldn't
we?" from
https://www.postgresql.org/message-id/CAKZiRmzo9xnJSgO4b26DTZqPuObcQ-6ncay%2BmOEKs9rzCkegUA%40mail.gmail.com