Skip site navigation (1) Skip section navigation (2)

Re: Server hitting 100% CPU usage, system comes to a crawl.

From: Brian Fehrle <brianf(at)consistentstate(dot)com>
To: Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com>
Cc: pgsql-general General <pgsql-general(at)postgresql(dot)org>
Subject: Re: Server hitting 100% CPU usage, system comes to a crawl.
Date: 2011-10-27 20:09:51
Message-ID: 4EA9BA8F.4010605@consistentstate.com (view raw or flat)
Thread:
Lists: pgsql-general
On 10/27/2011 01:48 PM, Scott Marlowe wrote:
> On Thu, Oct 27, 2011 at 12:39 PM, Brian Fehrle
> <brianf(at)consistentstate(dot)com>  wrote:
>> Looking at top, I see no SWAP usage, very little IOWait, and there are a
>> large number of postmaster processes at 100% cpu usage (makes sense, at this
>> point there are 150 or so queries currently executing on the database).
>>
>>   Tasks: 713 total,  44 running, 668 sleeping,   0 stopped,   1 zombie
>> Cpu(s):  4.4%us, 92.0%sy,  0.0%ni,  3.0%id,  0.0%wa,  0.0%hi,  0.3%si,
>>   0.2%st
>> Mem:  134217728k total, 131229972k used,  2987756k free,   462444k buffers
>> Swap:  8388600k total,      296k used,  8388304k free, 119029580k cached
> OK, a few points.  1: You've got a zombie process.  Find out what's
> causing that, it could be a trigger of some type for this behaviour.
> 2: You're 92% sys.  That's bad.  It means the OS is chewing up 92% of
> your 32 cores doing something.  what tasks are at the top of the list
> in top?
>
Out of the top 50 processes in top, 48 of them are postmasters, one is 
syslog, and one is psql. Each of the postmasters have a high %CPU, the 
top ones being 80% and higher, the rest being anywhere between 30% - 
60%. Would postmaster 'queries' that are running attribute to the sys 
CPU usage, or should they be under the 'us' CPU usage?


> Try running vmstat 10 for a a minute or so then look at cs and int
> columns.  If cs or int is well over 100k there could be an issue with
> thrashing, where your app is making some change to the db that
> requires all backends to be awoken at once and the machine just falls
> over under the load.

We've restarted the postgresql cluster, so the issue is not happening at 
this moment. but running a vmstat 10 had my 'cs' average at 3K and 'in' 
averaging around 9.5K.

- Brian F

In response to

Responses

pgsql-general by date

Next:From: Brian FehrleDate: 2011-10-27 20:15:25
Subject: Re: Server hitting 100% CPU usage, system comes to a crawl.
Previous:From: Scott MarloweDate: 2011-10-27 19:54:29
Subject: Re: Server hitting 100% CPU usage, system comes to a crawl.

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group