Re: Perf regression in 2.6.32 (Ubuntu 10.04 LTS)

On 09/13/2010 06:05 PM, Greg Smith wrote:
> Domas Mituzas wrote:
>> I've been playing around today a lot with sysbench, and observed that
>> 2.6.32 kernel supplied by Ubuntu is having perf regression with PG
>> (which does not affect MySQL), compared to 2.6.28 builds I have.
>> What I observed can be seen in a paste at
>> http://p.defau.lt/?8_GQV82Pz3_SDZbNOdP93Q (db12 is 2.6.28, db20 is
>> 2.6.32 - 2.6.32-24-server).
>> Machines are two socket quad-opterons 2356s.
>> oprofile output can be seen at
>> http://p.defau.lt/?OIR1vDFK4cze_fmBTQbV9w - system has >20% of idle
>> cpu, which is somewhere in the top symbol :)
>
> Are you using the same filesystem setup on both setups? And regardless,
> what is that filesystem? We know that between 2.6.28 and 2.6.32 the
> kernel improved how it handles fsync requests in a good way from a
> reliability perspective (to fix bugs that could cause data loss before),
> particularly on ext4, so it's possible the regression you're seeing is
> just the expense of handling things properly.
>
> If you already have sysbench on there, I'd suggest comparing the two
> systems by seeing how fast each can execute fsync requests:
>
> sysbench --test=fileio --file-fsync-freq=1 --file-num=1
> --file-total-size=16384 --file-test-mode=rndwr run | grep "Requests/sec"
>
> To help distinguish whether this regression might be coming from the
> already known changes in that area, or if it's instead from something
> that's impacting CPU efficiency.
>
> Also, it's easy to see a performance change of this size just from the
> database files being on a different part of the disk if you didn't
> control for that. Disks are almost twice as fast at their beginning than
> their end nowadays.

well the main point here is that domas is doing a pure read-only test on
a rather small workload so it should entirely fit in memory...
From some very quick testing here as well it rathers seems that for
some reason the CPU scheduler is not actually scheduling us all the
available CPU on 2.6.32 or we are having some sort of locking issue that
is more exposed on this kernel.

Stefan