Re: dump time increase by 1h with new kernel

Hi Everyone

On Fri, Oct 02, 2009 at 12:58:12PM -0700, Justin Pryzby wrote:
> When we upgraded from linux-2.6.24 to linux-2.6.27, our pg_dump
> duration increased by 20% from 5 hours to 6. My first attempt at

On Sat, Oct 03, 2009 at 11:31:11PM -0600, Scott Marlowe wrote:
> between these two to tell them exactly which one(s) causes the
> regression(s). That and how you compiled them or where they came from
These are both ubuntu kernels, and there's none in-between available
from their repository for testing. I could compile myself, but it
would have to include all the ubuntu patches (apparmor in
particular)..

On Thu, Oct 08, 2009 at 01:14:52AM -0400, Greg Smith wrote:
> report this in kernel land is "is it still slow on 2.6.[last
> stable|head]?".
I could try *newer* kernels, but not from newer ubuntu releases. This
machine is running ubuntu 8.04 with select packages from 8.10.
However it's running postgres 8.2, which isn't included with either of
those releases.. I tried dumping with 8.3 pg_dump, which had only
minimal effect.

> If you can try both kernel versions, the other thing you really
> should do is collect data from "vmstat 1" during the pg_dump period.
> It would help narrow what area is slower.
I have sar running on the machine, and came up with this:

07 and 30 are days of the month, 7 is last night, 30 is from
September. pg_dump starts around 9pm. tail -15 gets us to about
10pm. On sep 30, the machine was running 2.6.24 and the dump ran
until after 2am. Since we rebooted, it runs .27 and the dump runs
until after 4am. So the last column shows higher rate values for
every metric on the 30th (under .24) except for intr.

for a in user system iowait cswch tps rtps wtps intr; do for b in 07 30; do eval t$b='`sadf /var/log/sysstat/sa$b -- -A |grep -wi "$a" |tail -15 |awk "{sum+=\\$NF}END{print sum/NR}"`'; done; printf "%-6s %4.4s %4.4s %5.5s\n" $a $t30 $t07 `calc $t30/$t07`; done
s30 o07 s30/o07
user 13.9 6.85 ~2.03
system 0.56 0.37 ~1.52
iowait 0.61 0.52 ~1.16
cswch 873. 672. ~1.29
intr 121. 396. ~0.30
tps 412. 346. ~1.19
rtps 147. 143. ~1.02
wtps 264. 202. ~1.30

Not sure if sar can provide other data included by vmstat: IO merged
in/out, {,soft}irq ticks?

Thanks,
Justin