Re: XLogInsert scaling, revisited

On 20.09.2012 18:37, Tom Lane wrote:
> I suspect that it's not a coincidence that you're seeing
> a sweet spot for #slots ~= #CPUs.

Yeah, quite possible. I did not test with any different number of slots,
so I don't know if that's the sweet spot, but I wouldn't be surprised if
it is.

> If that is what's happening, we're going to want to be able to
> configure the #slots at postmaster start. Not sure how we'd go about
> it exactly - is there any reasonably portable way to find out how
> many CPUs the machine has? Or do we have to use a GUC for that?

Detecting the number of CPUs and using that might not be optimal. Even
with a machine with a lot of CPUs, a workload might not be limited by
WAL insertion speed. Perhaps we could have a counter of how often you
have to wait for a slot, and adjust the number of slots on the fly based
on that. Similar to the way the spinlock delay is adjusted.

At the moment, I'm grabbing the lock on a slot before determining which
blocks need to be backed up because of full_page_writes, and before
calculating the CRCs. I can try to move that so that the lock is grabbed
later, more like the WALInsertLock currently works. That would make the
duration the slot locks are held much shorter, which probably would make
the number of slots less important.

- Heikki