spinlock->pthread_mutex : real world results

Hi,

meanwhile we're using the patch in production (again, this is 9.1.3) and after
running it under full load for one week I believe it is pretty safe to say that
replacing the spinlock code with pthread_mutexes on Linux (which basically are a
futex wrapper) has solved the scalability issue and all stability/performance
problems on this system are simply gone.

While the improved pgbench run had already given a clear indication regarding
the optimization potential, we can now be pretty certain that spinlock
contention had really been the most significant root cause for the issues I had
described in my early postings ("why roll-your-own s_lock? / improving
scalability" / "experimental: replace s_lock spinlock code with pthread_mutex on
linux").

I am attaching annotated graphs showing the load averages and cpu statistics of
the respective machine. Please note the fact that the highest spikes have been
averaged out in these graphs. As I had mentioned before, with the original code
in place we had seen saturation of 64 cores and load averages in excess of 300.

I fully agree that improvements in more recent pgsql code to reduce the number
of required locks or, even better, lockless data structures are the way to go,
but for the remaining cases it should now have become apparent that favoring
efficient mutex implementations is advantageous for large SMPs, where they exist
(e.g. futexes on Linux).

Thanks, Nils