Skip site navigation (1) Skip section navigation (2)

Re: select on 22 GB table causes "An I/O error occured while sending to the backend." exception

From: david(at)lang(dot)hm
To: Florian Weimer <fweimer(at)bfk(dot)de>
Cc: henk de wit <henk53602(at)hotmail(dot)com>, pgsql-performance(at)postgresql(dot)org
Subject: Re: select on 22 GB table causes "An I/O error occured while sending to the backend." exception
Date: 2008-08-27 21:45:47
Message-ID: alpine.DEB.1.10.0808271436210.907@asgard.lang.hm (view raw or flat)
Thread:
Lists: pgsql-performance
On Wed, 27 Aug 2008, Florian Weimer wrote:

> * henk de wit:
>
>> On this table we're inserting records with a relatively low
>> frequency of +- 6~10 per second. We're using PG 8.3.1 on a machine
>> with two dual core 2.4Ghz XEON CPUs, 16 GB of memory and Debian
>> Linux. The machine is completely devoted to PG, nothing else runs on
>> the box.
>
> Have you disabled the OOM killer?

my understanding of the OOM killer is that 'disabling' it is disabling 
memory overcommit, making it impossible for you to get into a situation 
where the OOM killer would activate, but this means that any load that 
would have triggered the OOM killer will always start getting memory 
allocation errors before that point.

the OOM killer exists becouse there are many things that can happen on a 
system that allocate memory that 'may' really be needed, but also 'may 
not' really be needed.

for example if you have a process that uses 1G of ram (say firefox) and it 
needs to start a new process (say acroread to handle a pdf file), what it 
does is it forks the firefox process (each of which have 1G of ram 
allocated), and then does an exec of the acroread process (releasing the 
1G of ram previously held by that copy of the firefox process)

with memory overcommit enabled (the default), the kernel recognises that 
most programs that fork don't write to all the memory they have allocated, 
so it marks the 1G of ram that firefox uses as read-only, and if either 
copy of firefox writes to a page of memory it splits that page into 
seperate copies for the seperate processes (and if at this time it runs of 
of memory it invokes the OOM killer to free some space), when firefox does 
an exec almost immediatly after the fork it touches basicly none of the 
pages, so the process only uses 1G or ram total.

if memory overcommit is disabled, the kernel checks to see if you have an 
extra 1G of ram available, if you do it allows the process to continue, if 
you don't it tries to free memory (by throwing away cache, swapping to 
disk, etc), and if it can't free the memory will return a memroy 
allocation error (which I believe will cause firefox to exit).


so you can avoid the OOM killer, but the costs of doing so are that you 
make far less efficiant use of your ram overall.

David Lang

In response to

Responses

pgsql-performance by date

Next:From: Andrew SullivanDate: 2008-08-27 22:13:38
Subject: Re: select on 22 GB table causes "An I/O error occuredwhile sending to the backend." exception
Previous:From: Tom LaneDate: 2008-08-27 20:54:03
Subject: Re: Is there a way to SubPartition?

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group