Re: Server hitting 100% CPU usage, system comes to a crawl.

On Thu, Oct 27, 2011 at 12:39 PM, Brian Fehrle
<brianf(at)consistentstate(dot)com> wrote:
> Looking at top, I see no SWAP usage, very little IOWait, and there are a
> large number of postmaster processes at 100% cpu usage (makes sense, at this
> point there are 150 or so queries currently executing on the database).
>
>  Tasks: 713 total,  44 running, 668 sleeping,   0 stopped,   1 zombie
> Cpu(s):  4.4%us, 92.0%sy,  0.0%ni,  3.0%id,  0.0%wa,  0.0%hi,  0.3%si,
>  0.2%st
> Mem:  134217728k total, 131229972k used,  2987756k free,   462444k buffers
> Swap:  8388600k total,      296k used,  8388304k free, 119029580k cached

OK, a few points. 1: You've got a zombie process. Find out what's
causing that, it could be a trigger of some type for this behaviour.
2: You're 92% sys. That's bad. It means the OS is chewing up 92% of
your 32 cores doing something. what tasks are at the top of the list
in top?

Try running vmstat 10 for a a minute or so then look at cs and int
columns. If cs or int is well over 100k there could be an issue with
thrashing, where your app is making some change to the db that
requires all backends to be awoken at once and the machine just falls
over under the load.