Re: Moving more work outside WALInsertLock

On Thursday, December 15, 2011 02:51:33 PM Heikki Linnakangas wrote:
> I've been looking at various ways to make WALInsertLock less of a
> bottleneck on multi-CPU servers. The key is going to be to separate the
> two things that are done while holding the WALInsertLock: a) allocating
> the required space in the WAL, and b) calculating the CRC of the record
> header and copying the data to the WAL page. a) needs to be serialized,
> but b) could be done in parallel.
>
> I've been experimenting with different approaches to do that, but one
> thing is common among all of them: you need to know the total amount of
> WAL space needed for the record, including backup blocks, before you
> take the lock. So, here's a patch to move things around in XLogInsert()
> a bit, to accomplish that.
>
> This patch doesn't seem to have any performance or scalability impact. I
> must admit I expected it to give a tiny gain in scalability by
> shortening the time WALInsertLock is held by a few instructions, but I
> can't measure any. But IMO it makes the code more readable, so this is
> worthwhile for that reason alone.
Thats great! I did (or at least tried) something similar when I was playing
around with another crc32 implementation (which I plan to finish sometime). My
changes where totally whacky but I got rather big improvements when changing
the crc computation from incremental to one big swoop.
I started to hack up an api which buffered xlog data in statically sized buffer
in each backend and only submitted that every now and then. Never got that to
actually work correctly in more than the simplest cases though ;). In many
cases were taking the wal insert lock way to often during a single insert...
(you obviously know that...)

Andres