Skip site navigation (1) Skip section navigation (2)

Re: Hardware/OS recommendations for large databases (

From: Greg Stark <gsstark(at)mit(dot)edu>
To: stange(at)rentec(dot)com
Cc: Luke Lonergan <llonergan(at)greenplum(dot)com>, Greg Stark <gsstark(at)mit(dot)edu>, Dave Cramer <pg(at)fastcrypt(dot)com>, Joshua Marsh <icub3d(at)gmail(dot)com>, pgsql-performance(at)postgresql(dot)org
Subject: Re: Hardware/OS recommendations for large databases (
Date: 2005-11-21 19:01:26
Message-ID: 87ek59omex.fsf@stark.xeocode.com (view raw or flat)
Thread:
Lists: pgsql-performance
Alan Stange <stange(at)rentec(dot)com> writes:

> The point your making doesn't match my experience with *any* storage or program
> I've ever used, including postgresql.   Your point suggests that the storage
> system is idle  and that postgresql is broken because it isn't able to use the
> resources available...even when the cpu is very idle.  How can that make sense?

Well I think what he's saying is that Postgres is issuing a read, then waiting
for the data to return. Then it does some processing, and goes back to issue
another read. The CPU is idle half the time because Postgres isn't capable of
doing any work while waiting for i/o, and the i/o system is idle half the time
while the CPU intensive part happens.

(Consider as a pathological example a program that reads 8k then sleeps for
10ms, and loops doing that 1,000 times. Now consider the same program
optimized to read 8M asynchronously and sleep for 10s. By the time it's
finished sleeping it has probably read in all 8M. Whereas the program that
read 8k in little chunks interleaved with small sleeps would probably take
twice as long and appear to be entirely i/o-bound with 50% iowait and 50%
idle.)

It's a reasonable theory and it's not inconsistent with the results you sent.
But it's not exactly proven either. Nor is it clear how to improve matters.
Adding additional threads to handle the i/o adds an enormous amount of
complexity and creates lots of opportunity for other contention that could
easily eat all of the gains.

I also fear that heading in that direction could push Postgres even further
from the niche of software that works fine even on low end hardware into the
realm of software that only works on high end hardware. It's already suffering
a bit from that.

-- 
greg


In response to

Responses

pgsql-performance by date

Next:From: Alvaro HerreraDate: 2005-11-21 19:51:47
Subject: Re: Hardware/OS recommendations for large databases (
Previous:From: Luke LonerganDate: 2005-11-21 18:14:29
Subject: Re: Hardware/OS recommendations for large databases (

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group