Re: Trying out <stdatomic.h>

Hello,

On the earlier thread Tom asked (paraphrasing) what we'd actually be
buying and giving up by reimplementing port/atomics.h on top of C11
<stdatomic.h>, and whether the generated code and performance hold up
across the weird-and-wonderful platforms. So, I've tried to take
Thomas's WIP patches and put real numbers behind that question. I've
also made stdatomic opt-in at configure time.

Summary up front:

- It is opt-in and off by default: -Duse_stdatomic=no (the default)
keeps the traditional atomics, which are untouched. This is an
alternative implementation, not a replacement at this time.
- There is *no* measurable regression on x86-64.
- Functionally it works and is tested correct on x86-64, aarch64,
RISC-V, FreeBSD, and Windows-on-ARM (MSVC).
- Requires a working C11 <stdatomic.h>: MSVC 2022+ (with
/experimental:c11atomics), GCC 4.9+/Clang 3.1+. Anything older
just builds the traditional path.

The series (v4 on 7b87f08e877 cf-7005) is three patches:

1. The whole implementation -- build-time detection, the C11 code,
and the wire-up. This is the only patch proposed for commit, and
it is opt-in and off by default (-Duse_stdatomic / --with-stdatomic
= auto|yes|no, default no).
2. (not for merge) the benchmark harness, included for transparency.
3. (not for merge, for discussion) removes the traditional
implementation entirely, to scope that eventual removal and
ensure that CI that tests this patch set is testing stdatomics
not exiting atomics.

Per-operation mapping
---------------------
Every hand-written pg_atomic entry point and how it maps to C11. "trad"
is what the existing arch-*/generic-* code effectively provides; the
mapping column is the atomic_*_explicit order this series emits.

operation | trad semantics | stdatomic map | notes
-------------------------+-------------------+---------------+------
read_u32/u64 | plain, no barrier | seq_cst load | (1)
write_u32/u64 | plain, no barrier | relaxed store | (1)
unlocked_write_u32/u64 | plain, no atomic | relaxed store | (2)
read_membarrier_u32/u64 | read + full fence | seq_cst load |
write_membarrier_u32/u64 | write + full fence| seq_cst store |
init_u32/u64 | plain store | atomic_init | (3)
-------------------------+-------------------+---------------+------
exchange_u32/u64 | full barrier | seq_cst xchg |
compare_exchange_u32/u64 | full barrier | seq_cst CAS | (4)
-------------------------+-------------------+---------------+------
fetch_add_u32/u64 | full barrier | seq_cst RMW |
fetch_sub_u32/u64 | full barrier | seq_cst RMW |
fetch_and_u32/u64 | full barrier | seq_cst RMW |
fetch_or_u32/u64 | full barrier | seq_cst RMW |
add_fetch_u32/u64 | full barrier | RMW + add | (5)
sub_fetch_u32/u64 | full barrier | RMW - sub | (5)
monotonic_advance_u64 | CAS loop | CAS loop | (6)
-------------------------+-------------------+---------------+------
init_flag | store 0/unlocked | atomic_init | (7)
test_set_flag | TAS, acquire | fetch_and,acq | (7)
unlocked_test_flag | plain read | relaxed load | (7)
clear_flag | store, release | release store | (7)
-------------------------+-------------------+---------------+------
pg_compiler_barrier | compiler barrier | signal_fence | (8)
pg_read_barrier | acquire barrier | signal+acquire| (8)
pg_write_barrier | release barrier | signal+release| (8)
pg_memory_barrier | full barrier | signal+seq_cst| (8)
pg_spin_delay | pause/isb/yield | unchanged |

(1) The crux of this mail -- see "read/write ordering" below. The
write is relaxed (plain STR, matching the "no barrier" contract);
the read is seq_cst, which weak-memory correctness required.
(2) unlocked_write is a plain non-atomic store when exclusive access
is guaranteed; relaxed matches.
(3) init is not atomic; only legal before publication.
(4) strong CAS, seq_cst on both success and failure paths.
(5) C11 has no add_fetch/sub_fetch; compose from fetch_* + arith.
(6) unchanged; built on compare_exchange in atomics.h, no impl needed.
(7) flag is backed by a 32-bit word, not uint8 -- see "flag width".
Internal polarity is inverted (1=unlocked) but the public
contract (test_set returns true on acquire) is identical.
(8) a bare atomic_thread_fence does not by itself constrain the
compiler's movement of plain (non-atomic) accesses, so each
barrier is atomic_signal_fence (compiler barrier) *plus* a
thread fence -- see "barriers".

What the standard atomics do differently, under the covers
----------------------------------------------------------
Mostly this is renaming our names to the standard ones. The
non-obvious differences that matter:

* Barriers. A bare atomic_thread_fence() does not by itself constrain
the compiler from moving plain, non-atomic accesses across it; our
pg_read/write/memory_barrier() must order those too. So each barrier
is a compiler barrier (atomic_signal_fence) *plus*
atomic_thread_fence(acquire/release/seq_cst). Miss the compiler
barrier and the optimizer will happily reorder non-atomic
loads/stores across it -- I found this the hard way (lost tuples in
parallel hash join on RISC-V; more below).

* Flag width. Standard atomic_flag has no relaxed load, so like
Thomas's patch I build pg_atomic_flag on an integer. It must be
32-bit, not 8-bit: RISC-V has no byte-granular AMO, so an 8-bit
fetch_and is emulated as a read-modify-write of the whole containing
word, which clobbers adjacent bytes in a packed slock_t. (generic.h
already uses uint32 for exactly this reason.) Note that Heikki's
in-flight "Replace pg_atomic_flag with pg_atomic_bool" patch removes
pg_atomic_flag entirely; I'll rebase onto pg_atomic_bool rather than
reimplement the flag, which also makes the polarity note (7) moot.

* RMW ops (fetch_add, compare_exchange, exchange) are seq_cst in both
the old and new worlds, so those are unchanged.

* Lock-free, asserted. PostgreSQL's atomics live in shared memory and
are touched from multiple processes, not just threads. A
non-lock-free _Atomic that the compiler lowers to a libatomic lock
table would use process-local locks and silently corrupt shared
state. So this asserts at compile time, per width, that the atomics
are genuinely lock-free
(StaticAssertDecl(ATOMIC_{CHAR,SHORT,INT,LLONG}_LOCK_FREE == 2, ...)),
as Thomas's patch did; dropping those would be a footgun specific to
our cross-process usage. Relatedly, the configure/meson probe is a
*link* test with a -latomic fallback, not just a compile test: a
64-bit compare-exchange can lower to a libatomic call
(__atomic_compare_exchange_8) on some 32-bit targets, where a
compile-only check would pass and then fail at final link.

read/write ordering: the interesting mistake
---------------------------------------------
This is where I changed my mind, and it's the part worth reviewing.

pg_atomic_read_u32/write_u32 have always been documented as plain
atomic (non-torn) accesses with *no barrier semantics*. On ARM that
is exactly LDR/STR; on RISC-V a plain lw/sw. The faithful C11 mapping
of that contract is memory_order_relaxed. That is right for the write.
It turned out NOT to be safe for the read on weak-memory hardware, and
my first cut got the fix backwards -- I strengthened both when only the
read needed it.

My first cut shipped seq_cst for read/write. The reason was a real
correctness failure: on RISC-V, a parallel hash join intermittently
lost a tuple (join_hash "extremely_skewed", 19999 instead of 20000
rows), stdatomic build only. Bisected on the hardware: relaxed fails,
seq_cst passes 10/10. seq_cst made the symptom go away, so I shipped
it -- as a blanket default on the generic read/write.

That was a blunt over-fix, and the benchmarks are the evidence. On
aarch64, atomic_store_explicit(seq_cst) compiles to STLR (store-
release); a plain write is STR. Under cross-core contention on a hot
line (LWLock state on release, buffer header state), STLR drains the
store buffer and joins the total store order, serializing writers. A
contended-write microbench shows the shape starkly (yes = stdatomic,
threads hammering one line):

threads yes ns no ns ratio
2 0.427 0.256 1.67x
4 0.392 0.130 3.02x
8 0.482 0.103 4.68x
16 0.471 0.099 4.76x

Uncontended, single-threaded, they're identical (write_u32 0.511 vs
0.509 ns; read_u32 0.919 vs 0.916 ns) -- LDAR/STLR in isolation cost
nothing here. The cost is purely the store-ordering under contention.
Read-only pgbench is dense in exactly those hot atomic writes on lock
acquire/release, so the microscopic per-op difference compounded into
the end-to-end ~2% below. x86 is immune: on TSO a seq_cst load is a
plain mov and a store is cheap, so read/write ordering is nearly free
regardless. In other words the 2% and the 4.8x are both symptoms of
one wrong decision: strengthening a read/write path whose documented
contract never asked for it.

The right question is whether the write and the read need the same
treatment. They do not. I bisected the read on RISC-V (join_hash,
with the compiler+thread-fence barrier fix described above already in
place):

read variant join_hash
traditional volatile load (stock) 20/20 PASS
C11 relaxed atomic load 2/20 (broken)
C11 relaxed load + compiler barrier 4/20 (broken)
C11 seq_cst atomic load 15/15 PASS
(all with write relaxed; acquire read also failed, 2/12.)

The surprise is the top two rows: the traditional volatile load passes
20/20, but a C11 memory_order_relaxed load of the same variable fails
2/20 -- on the same hardware, with identical barriers. They are not
equivalent. A volatile access can't be reordered by the compiler
relative to the surrounding pg_read_barrier()/spinlock, so those
barriers actually constrain it; a relaxed atomic the compiler may hoist
or sink across them, defeating the barrier at the consumer. Wrapping
the relaxed load in a compiler barrier isn't enough either (4/20).
Only seq_cst reproduces what the volatile load gave us. So the write
goes back to relaxed (plain STR, matching stock's "no barrier"
contract and erasing the contended-store penalty above), and the read
stays seq_cst.

I want to be honest that I first tried the narrower thing, because it's
the obvious objection: return the generic read to relaxed (its
documented "no barrier" contract) and instead order the one parallel
hash-join consumer that lost the tuple -- a pg_read_barrier() in
ExecParallelHashFirstTuple() between reading the bucket head and
dereferencing the tuple it points to. On RISC-V that did NOT work:
join_hash still lost a tuple, about 1 run in 4, the same as relaxed
with no consumer barrier at all:

read variant join_hash
C11 relaxed load + pg_read_barrier() at the
hash-join bucket-read consumer fails (~1 in 4)

The lesson is that the ordering the seq_cst read provides is not
localized to that one bucket read. The parallel hash join reaches
shared state through the Barrier / condition-variable / LWLock
machinery as well, and all of those do pg_atomic_read_u32() and were
quietly relying on the compiler-ordering the traditional *volatile*
load gave for free. Relaxing the generic read loses that property
everywhere at once; re-establishing it consumer-by-consumer would mean
auditing and barriering many call sites across lmgr/ipc/executor -- a
much larger and riskier change than this port. So the read stays
seq_cst: it is the single change that restores, tree-wide, exactly the
property the volatile load provided, and -- per the numbers below --
it costs nothing measurable end-to-end.

On the memory-model shape: a seq_cst read paired with a relaxed write
does not by itself establish a single total order over read and write.
The read's seq_cst is there specifically to restore the
compiler-ordering property the old volatile read had, which is what the
consumers depended on -- not to claim a read/write total order. If
someone sees how to return the generic read to relaxed without a
tree-wide consumer audit, I would genuinely like to know; I could not
find one that survives RISC-V.

Re-benchmark with read=seq_cst + write=relaxed, RO pgbench,
backends pinned to one NUMA node, clients on the other, 8 x 45s runs
per point, A/B alternated, median tps, Mann-Whitney p:

clients aarch64 ratio (p) x86 ratio (p)
1 1.023 (0.009) 1.013 (0.002)
4 1.007 (0.021) 1.000 (0.916)
16 1.013 (0.001) 1.002 (0.074)
48 1.020 (0.001) 1.007 (0.002)
96 1.000 (0.529) 1.002 (0.141)

(ratio = stdatomic / stock; >1 means stdatomic faster.) The ~2%
aarch64 regression is gone -- stdatomic is now at parity-to-slightly-
faster on both arches. The only sub-1.0 cells are at 192 clients,
where backends are oversubscribed past the one node's core count; that
is placement noise, not an atomics signal.

Test methodology
----------------
Two bare-metal EC2 instances, 192 vCPU / 2 NUMA nodes each: Intel
Sapphire Rapids (m7i.metal-48xl) and Graviton5 (c9g.metal-48xl). Turbo
off, governor=performance, THP off, numa_balancing off, explicit
hugepages, irqbalance off. Both variants built from one checkout, same
compiler/flags, only -Duse_stdatomic differing, buildtype=release,
cassert=off. Dataset cache-resident (scale 100, shared_buffers
32-64GB) so I measure the lock/atomic paths, not I/O; fsync=off,
synchronous_commit=off for the same reason. pgbench -M prepared, -S
read-only and -N write. Clients swept 1..256; 10 x 60s measured runs
per (variant, clients, workload), A/B alternated to cancel drift;
median, MAD, ratio with bootstrap 95% CI, and Mann-Whitney U. Plus a
standalone microbench linked against the installed port/atomics.h that
times each pg_atomic_* op directly, uncontended and contended.

NUMA placement. For consistent numbers I pinned all backends to NUMA
node 0 and pgbench to node 1. That deliberately holds the server on a
single node, so sweeping clients past one node's physical core count
measures core oversubscription on node 0, not NUMA-spanning scaling --
the high-client "knee" cells are that oversubscription, not a memory-
locality effect. I also ran a cross-node placement (backends
interleaved across both nodes with numactl --interleave=all, pgbench
floating) to check for a stdatomic-specific NUMA interaction. Result:
none. Interleaving makes both variants slower in absolute terms (more
cross-node cache traffic, as expected), and the stdatomic/stock ratio
under interleave swings both directions across client counts and is
dominated by placement and oversubscription noise, not by any
consistent stdatomic penalty. That matches the mechanism -- the
seq_cst read cost is a per-core store-order effect, not a cross-socket
coherence effect -- so I don't see a NUMA angle here, which is what I
expected but wanted to confirm rather than assume.

Numbers
-------
x86-64, read-only and write: parity. Every cell within +/-0.5% of
stock, MAD ~0.1%. A few cells cross p<0.05 only because the variance
is so small; the effect is in the noise floor and both directions
appear. As expected on TSO.

aarch64 read-only, first cut (blanket seq_cst read/write) -- consistent
regression, tight CI, p<0.001:

clients yes tps no tps ratio 95% CI
1 74,227 75,127 0.988 [0.980, 0.992]
4 286,689 291,934 0.982 [0.977, 0.985]
16 1,126,114 1,143,962 0.984 [0.982, 0.987]
192 3,563,812 3,624,204 0.983 [0.983, 0.984]
256 3,525,938 3,599,808 0.979 [0.978, 0.981]

i.e. ~1.5-2% slower across the board (c=64/96 invert; that's the
oversubscription knee, high MAD, not signal). aarch64 write (-N) is
within noise.

aarch64 read-only, the fix (seq_cst read + relaxed write) -- re-bench,
backends on node 0, clients on node 1, 8 x 45s runs/point, ratio =
stdatomic/stock, Mann-Whitney p:

clients yes tps no tps ratio p
1 76,931 75,195 1.023 0.009
4 291,624 289,523 1.007 0.021
16 1,142,676 1,128,195 1.013 0.001
48 3,117,124 3,056,956 1.020 0.001
96 3,406,632 3,408,343 1.000 0.529

The regression is gone: stdatomic is now parity-to-slightly-faster on
aarch64, because the write is back to a plain STR while the read's
seq_cst LDAR costs nothing measurable here. (The 192-client cell dips
below 1.0 on both arches, but that's backends oversubscribed past one
node's core count, not an atomics effect.)

Caveats on the data: Intel is complete (RO+RW, 10 runs/cell).
Graviton5 RO is complete; the Graviton5 -N sweep was re-run separately
with per-cell table re-init (autovacuum-off tpcb bloats across runs)
and is parity/noise. I have the raw CSVs, per-cell stats, and
microbench data and can share them. The cross-node NUMA study
(interleaved placement) is described above; I can post those numbers
too if useful.

I'll post the rebased series (on PG master) as attachments in a
follow-up if there's interest. The correctness findings apply to any
<stdatomic.h> port: pair every thread fence with a compiler barrier;
make pg_atomic_flag 32-bit; and -- the lesson above -- a C11
memory_order_relaxed load is *not* the same as stock's volatile load.

What this series ships is the split the hardware forced: the generic
WRITE is relaxed (plain STR, matching the "no barrier" contract and
erasing the contended-store penalty), and the generic READ is seq_cst,
which is what RISC-V actually required (15/15) and which -- because the
expensive part was the store, now relaxed -- costs nothing measurable
end-to-end. I tried the narrower "relax the read, order the one
consumer" alternative and it did not hold up on RISC-V (above), so I am
not proposing it; if there's a way to get the generic read back to
relaxed without a tree-wide consumer audit, I'd welcome it.

best,

-greg