Re: Some interesting results from tweaking spinlocks

Your observation that spinning instead of sleeping being faster on SMP makes
sense.

On a single processor system, if you don't have the lock, you should call
select() as soon as possible (never spin). This will allow the OS (presumably)
to switch to the process who does. You will never get the lock unless your
process loses the CPU because some other process MUST get CPU time in order to
release the lock.

On an SMP machine, this is different, other processes can run truly
simultaneously to the process spinning. Then you have the trade-off of wasting
CPU cycles vs sleeping.

A better lock system could know how many CPUs are in a system, and how many
processes are waiting for the lock. Use this information to manage who sleeps
and who spins.

For instance, if you have a 2 CPU SMP box, the first process to get the lock
gets it. The next process to try for the lock should spin. The third process
waiting should sleep.

ATOMIC_INC(lock->waiters);

while(TAS(lock))
{
if (++delays > (TIMEOUT_MSEC / DELAY_MSEC))
s_lock_stuck(lock, file, line);
if(lock->waiters >= num_cpus)
{
delay.tv_sec = 0;
delay.tv_usec = DELAY_MSEC * 1000;
(void) select(0, NULL, NULL, NULL, &delay);
}
}

ATOMIC_DEC(lock->waiters);

The above code is probably wrong, but something like it may improve performance
on SMP and uniprocessor boxes. On a uniprocessor box, the CPU is released right
away on contention. On an SMP box light contention allows some spinning, but on
heavy contention the CPUs aren't wasting a lot of time spinning.