Re: greenfly lwlock corruption in REL_14_STABLE and REL_15_STABLE

Hello hackers,

12.12.2025 02:54, Thomas Munro wrote:
> Thanks! Could also be environmental. Maybe the snow arrived and you
> adjusted your thermostat? :-) But even then we'd still need to figure
> out what's different on those branches... One difference is autoconf
> vs meson. Both say -g -O2, but only Meson says -fPIC. Can you
> reproduce it on later branches with configure/make?

I'd like to revive this thread, while we have no snow in EU yet. We had
some off-list discussion of this topic at the end of the past year, but
didn't root-cause the issue then. But this week I have finally dedicated
some time to dive into this with my Orange Pi device, which is able to
reproduce those crashes more or less as often as greenfly.

I came to the following modification:
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -4250,3 +4250,10 @@ LockBuffer(Buffer buffer, int mode)
        else if (mode == BUFFER_LOCK_EXCLUSIVE)
+{
+for (int i = 0; i < 100; i++)
+{
+LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+LWLockRelease(BufferDescriptorGetContentLock(buf));
+}
                LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+}
        else

which makes a build produced on REL_15_STABLE with:
CC=clang-20 CFLAGS="-O1" ./configure -q --enable-debug --enable-cassert --enable-tap-tests && make -s -j8

fail for me reliably (0 of 10 runs succeeded) as below:
PROVE_TESTS="t/001*" make -s check -C src/bin/pgbench
...
Files=1, Tests=3,  9 wallclock secs ( 0.11 usr  0.03 sys +  4.83 cusr  1.02 csys =  5.99 CPU)
Result: FAIL

TRAP: FailedAssertion("!(oldstate & LW_VAL_EXCLUSIVE)", File: "lwlock.c", Line: 1843, PID: 64342)
postgres: main: orangepi postgres [local] CREATE TYPE(ExceptionalCondition+0x72)[0x2ab2066546]
postgres: main: orangepi postgres [local] CREATE TYPE(LWLockRelease+0x58e)[0x2ab2385160]
postgres: main: orangepi postgres [local] CREATE TYPE(_bt_relbuf+0x14)[0x2ab20d5060]
postgres: main: orangepi postgres [local] CREATE TYPE(+0x13bc2a)[0x2ab20d1c2a]
...
(other observed result is a hang, once per ~10 runs)

Please find attached the assembler code for the LWLockWakeup() function
that is known to produce the assertion failure and note two instructions
I commented there:
  3eee4a:       lw a3,4(s6)                                        # 1014: old_state = pg_atomic_read_u32(&lock->state)
  3eee4e:       lw a5,8(s6)                                        # 1026: proclist_is_empty(&lock->waiters)

The following minimal modification:
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -1016,2 +1016,3 @@ LWLockWakeup(LWLock *lock)
                {
+__asm__ __volatile__("nop" ::: "memory");
                        desired_state = old_state;

which turns that pair into:
  3eee4a:       lw      a3,4(s6)
  3eee4e:       nop
  3eee50:       lw      a5,8(s6)

makes the test pass reliably for me (10+ iterations).

So it's not surprising now that this crash depends on postgres version
(v14/v15 only), compiler (gcc doesn't produce the code needed), its'
options (not reproduced with -O0), etc.

I've generated a minimal standalone reproducer for this anomaly (please
find it attached too), which replicates an essential part of
LWLockWakeup():
      sb      pending_wakeup, (waiting)
      lw      old_state, 4(lock)
      lw      head, 8(lock)
      and     desired_state, old_state, mask
  1:  lr.w    cur, (state)
      bne     cur, old_state, 2f
      sc.w    tmp, desired_state, (state)
      bnez    tmp, 1b
  2:

and demonstrates that LR/SC mechanism doesn't work as expected at least on
the given RISC-V CPU (8 SpacemiT X60 cores in 2 clusters):
Machine model: ky x1 orangepi-rv2 board
...
processor       : 7
hart            : 7
model name      : Ky(R) X1
isa             :
rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnapot_svpbmt
mmu             : sv39
uarch           : ky,x60
mvendorid       : 0x710
marchid         : 0x8000000058000001
mimpid          : 0x1000000049772200

Let me share results of a few experiments:
clang-20 -O1 ~/lrsc-anomaly.c -o ~/lrsc-anomaly
~/lrsc-anomaly
LOST UPDATE in round 28: value=0x70001d01 expected=0x00001d01
LOST UPDATE in round 29: value=0x70001e01 expected=0x00001e01
LOST UPDATE in round 30: value=0x70001f01 expected=0x00001f01
LOST UPDATE in round 31: value=0x70002001 expected=0x00002001
LOST UPDATE in round 32: value=0x70002101 expected=0x00002101
LOST UPDATE in round 33: value=0x70002201 expected=0x00002201
LOST UPDATE in round 34: value=0x70002301 expected=0x00002301
LOST UPDATE in round 35: value=0x70002401 expected=0x00002401
LOST UPDATE in round 37: value=0x70002601 expected=0x00002601
LOST UPDATE in round 38: value=0x70002701 expected=0x00002701
page_offset=0 rounds=1000000 retried=1824 lost=15523: FAILED

It reproduces the issue: process A's SC succeeds despite B's increment
happened in between.

for n in `seq 0 15`; do ~/lrsc-anomaly -n $n | tail -1; done
page_offset=0 rounds=1000000 retried=5374 lost=16160: FAILED
page_offset=1 rounds=1000000 retried=29563 lost=16438: FAILED
page_offset=2 rounds=1000000 retried=1251 lost=17209: FAILED
page_offset=3 rounds=1000000 retried=14960 lost=22: FAILED
page_offset=4 rounds=1000000 retried=12090 lost=19: FAILED
page_offset=5 rounds=1000000 retried=13561 lost=15: FAILED
page_offset=6 rounds=1000000 retried=15037 lost=32: FAILED
page_offset=7 rounds=1000000 retried=3124 lost=16774: FAILED
page_offset=8 rounds=1000000 retried=19813 lost=0: OK
page_offset=9 rounds=1000000 retried=20210 lost=0: OK
page_offset=10 rounds=1000000 retried=17992 lost=0: OK
page_offset=11 rounds=1000000 retried=18207 lost=0: OK
page_offset=12 rounds=1000000 retried=17903 lost=0: OK
page_offset=13 rounds=1000000 retried=18779 lost=0: OK
page_offset=14 rounds=1000000 retried=18384 lost=0: OK
page_offset=15 rounds=1000000 retried=18328 lost=0: OK

That is, the effect depends not only on that lw pair, but also on offset
of "waiting" in "sb pending_wakeup, (waiting)" preceding it.

The anomaly's probability also depends on whether these two cores/harts
are in the same cluster or not. For example, with cores 0 and 1 that belong
to cluster 0:
for n in `seq 0 4 15`; do taskset -c 0,1 ~/lrsc-anomaly -n $n | tail -1; done
page_offset=0 rounds=1000000 retried=1194 lost=16909: FAILED
page_offset=4 rounds=1000000 retried=1147 lost=16464: FAILED
page_offset=8 rounds=1000000 retried=18658 lost=0: OK
page_offset=12 rounds=1000000 retried=18154 lost=0: OK

And now with two cores that belong to different clusters:
for n in `seq 0 4 15`; do taskset -c 0,4 ~/lrsc-anomaly -n $n | tail -1; done
page_offset=0 rounds=1000000 retried=13350 lost=45: FAILED
page_offset=4 rounds=1000000 retried=15566 lost=37: FAILED
page_offset=8 rounds=1000000 retried=18223 lost=0: OK
page_offset=12 rounds=1000000 retried=17466 lost=0: OK

Also, nop inserted as above:
                /* old_state = pg_atomic_read_u32(&lock->state) */
                "       lw              %[old_state], 4(%[lock])\n"
+"nop\n"
                /* proclist_is_empty(&lock->waiters) */
                "       lw              %[head], 8(%[lock])\n"

changes the picture:
for n in `seq 0 15`; do taskset -c 0,1 ~/lrsc-anomaly -n $n | tail -1; done
page_offset=0 rounds=1000000 retried=15306 lost=17557: FAILED
page_offset=1 rounds=1000000 retried=2624 lost=16711: FAILED
page_offset=2 rounds=1000000 retried=1468 lost=16940: FAILED
page_offset=3 rounds=1000000 retried=815 lost=17098: FAILED
page_offset=4 rounds=1000000 retried=31171 lost=0: OK
page_offset=5 rounds=1000000 retried=17372 lost=0: OK
page_offset=6 rounds=1000000 retried=31803 lost=0: OK
page_offset=7 rounds=1000000 retried=19194 lost=0: OK
page_offset=8 rounds=1000000 retried=5777 lost=0: OK
page_offset=9 rounds=1000000 retried=5375 lost=0: OK
page_offset=10 rounds=1000000 retried=11920 lost=0: OK
page_offset=11 rounds=1000000 retried=5334 lost=0: OK
page_offset=12 rounds=1000000 retried=5087 lost=0: OK
page_offset=13 rounds=1000000 retried=12040 lost=0: OK
page_offset=14 rounds=1000000 retried=8479 lost=0: OK
page_offset=15 rounds=1000000 retried=4965 lost=0: OK

It kind of confirms the behavior observed with Postgres, when nop shifted
the code out of the problem area.

It also depends on number of distinct lines (buffers) in rotation; with
#define NBUFFERS        2

the probability drops almost to zero:
taskset -c 0,1 ~/lrsc-anomaly
LOST UPDATE in round 300830: value=0x7000ba01 expected=0x0000ba01
LOST UPDATE in round 300831: value=0x7000bb01 expected=0x0000bb01
page_offset=0 rounds=1000000 retried=2 lost=2: FAILED

Sorry for the long scroll, but it looks like this anomaly should be added
to our collection of the outer world's defects discovered during Postgres
testing.

Best regards,
Alexander