Re: buffer assertion tripping under repeat pgbench load

On Thu, Dec 27, 2012 at 3:17 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> The thing that this theory has a hard time with is that the buffer's
> global refcount is zero. If you assume that there's a bit that
> sometimes randomly goes to 1 when it should be 0, then what I'd expect
> to typically happen is that UnpinBuffer sees nonzero LocalRefCount and
> hence doesn't drop the session's global pin when it should. The only
> way that doesn't happen is if decrementing LocalRefCount to zero stores
> a nonzero pattern when it should store zero, but nonetheless the CPU
> thinks it stored zero.

It seems entirely plausible when you factor in the L2 cache. The cpu
could be happily incrementing and decrementing the count entirely
correctly blissfully unaware that the value being stored in the DRAM
has this extra bit set every time. Not until the transaction ends and
it has to refetch the cache line because enough time has passed for it
to age out of L2 cache does it find the corrupt value.

> At the moment I like the other theory you alluded to, that this is a
> wild store from code that thinks it's manipulating some other data
> structure entirely. The buffer IDs will help confirm or refute that
> perhaps. No idea ATM how we would find the problem if it's like that

Yeah, though I have to admit the particular data structures I
mentioned are entirely irrelevant. Looking back at the code
LocalRefCount is heap allocated and BufferBlocks is actually a pointer
to shmem.

If it's always the first buffer then it could conceivably still be
some other heap allocated object that always lands before
LocalRefCount. It does seem a bit weird to be storing 1<<30 though --
there are no 1<<30 constants that we might be storing for example.

It occurs to me that it should be possible to walk the list of
resource owners and count up what the correct LocalRefCount should be
every time we increment or decrement it. That should catch the error
sooner -- if it's not a hardware problem that the cache is hiding.

--
greg