From: | Lonni J Friedman <netllama(at)gmail(dot)com> |
---|---|
To: | Kevin Grittner <kgrittn(at)ymail(dot)com> |
Cc: | Jeff Janes <jeff(dot)janes(at)gmail(dot)com>, Eduardo Morras <emorrasg(at)yahoo(dot)es>, pgsql-general <pgsql-general(at)postgresql(dot)org> |
Subject: | Re: upgrade from 9.2.x to 9.3 causes significant performance degradation |
Date: | 2013-09-18 17:30:16 |
Message-ID: | CAP=oouH0tXbA-4qw0Hd0tbJV2wj1Lp8A=UaSf-dvV2-soap5jA@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-general |
On Wed, Sep 18, 2013 at 2:02 AM, Kevin Grittner <kgrittn(at)ymail(dot)com> wrote:
> Lonni J Friedman <netllama(at)gmail(dot)com> wrote:
>
>> top shows over 90% of the load is in sys space. vmstat output
>> seems to suggest that its CPU bound (or bouncing back & forth):
>
> Can you run `perf top` during an episode and see what kernel
> functions are using all that CPU?
I take back what I said earlier. While the master is currently back
to normal performance, the two hot standby slaves are still churning
something awful.
If I run 'perf top' on either slave, after a few seconds, these are
consistently the top three in the list:
84.57% [kernel] [k] _spin_lock_irqsave
6.21% [unknown] [.] 0x0000000000659f60
4.69% [kernel] [k] compaction_alloc
>
> This looks similar to cases I've seen of THP defrag going wild.
> Did the OS version or configuration change? Did the PostgreSQL
> memory settings (like shared_buffers) change?
I think you're onto something here with respect to THP defrag going
wild. I set /sys/kernel/mm/transparent_hugepage/defrag to 'never' and
immediately the load dropped on both slaves from over 5.00 to under
1.00.
So this raises the question, is this a kernel bug, or is there some
other solution to the problem?
Also, seems weird that the problem didn't happen until I switched from
9.2 to 9.3. Is it possible this is somehow related to the change from
using SysV shared memory to using Posix shared memory and mmap for
memory management?
From | Date | Subject | |
---|---|---|---|
Next Message | Jeff Janes | 2013-09-18 18:00:59 | Re: upgrade from 9.2.x to 9.3 causes significant performance degradation |
Previous Message | Jeff Janes | 2013-09-18 17:25:20 | Re: Query - CPU issue |