Skip site navigation (1) Skip section navigation (2)

Linux/PostgreSQL scalability issue - problem with 8 cores

From: Jakub Ouhrabka <kuba(at)comgate(dot)cz>
To: pgsql-performance(at)postgresql(dot)org
Subject: Linux/PostgreSQL scalability issue - problem with 8 cores
Date: 2008-01-03 11:50:01
Message-ID: flii57$748$1@news.hub.org (view raw or flat)
Thread:
Lists: pgsql-performance
Hi all,

we have a PostgreSQL dedicated Linux server with 8 cores (2xX5355). We 
came accross a strange issue: when running with all 8 cores enabled 
approximatly once a minute (period differs) the system is very busy for 
a few seconds (~5-10s) and we don't know why - this issue don't show up 
when we tell Linux to use only 2 cores, with 4 cores the problem is here 
but it is still better than with 8 cores - all on the same machine, same 
config, same workload. We don't see any apparent reason for these peaks. 
We'd like to investigate it further but we don't know what to try next. 
Any suggenstions? Any tunning tips for Linux+PostgreSQL on 8-way system? 
Can this be connected with our heavy use of listen/notify and hundreds 
backends in listen mode?

More details are below.

Thanks,

Kuba

System: HP DL360 2x5355, 8G RAM, P600+MSA50 - internal 2x72GB RAID 10 
for OS, 10x72G disks RAID 10 for PostgreSQL data and wal
OS: Linux 2.6 64bit (kernel 2.6.21, 22, 23 makes little difference)
PostgreSQL: 8.2.4 (64bit), shared buffers 1G

Nothing else than PostgreSQL is running on the server. Cca 800 
concurrent backends. Majority of backends in LISTEN doing nothing. 
Client interface for most backends is ecpg+libpq.

Problem description:

The system is usually running 80-95% idle. Approximatly once a minute 
for cca 5-10s there is a peak in activity which looks like this:

vmstat (and top or atop) reports 0% idle, 100% in user mode, very low 
iowait, low IO activity, higher number of contex switches than usual but 
not exceedingly high (2000-4000cs/s, usually 1500cs/s), few hundreds 
waiting processes per second (usually 0-1/s). From looking at top and 
running processes we can't see any obvious reason for the peak. 
According to PostgreSQL log the long running commands from these moments 
are e.g. begin transaction lasting several seconds.

When only 2 cores are enabled (kernel command line) then everything is 
running smoothly. 4 cores exibits slightly better behavior than 8 cores 
but worse than 2 cores - the peaks are visible.

We've tried kernel versions 2.6.21-23 (latest revisions as of beginning 
December from kernel.org) the pattern slightly changed but it may also 
be that the workload slightly changed.

pgbench or any other stress testing runs smoothly on the server.

The only strange thing about our usage pattern I can think of is heavy 
use of LISTEN/NOTIFY especially hunderds backends in listen mode.

When restarting our connected clients the peaks are not there from time 
0, they are visible after a while - seems something gets synchronized 
and causing troubles then.

Since the server is PostgreSQL dedicated and no our client applications 
are running on it - and there is a difference when 2 and 8 cores are 
enabled -  we think that the peaks are not caused by our client 
applications.

How can we diagnose what is happening during the peaks?

Responses

pgsql-performance by date

Next:From: Peter ChildsDate: 2008-01-03 14:35:37
Subject: Commit takes a long time.
Previous:From: Sebastián BaioniDate: 2008-01-02 12:24:03
Subject: Re: [GENERAL] Can't make backup

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group