Re: "Too many open files in system" error

"ulimit -Hn" shows 11095 which is the same as kern.maxfilesperproc.

"ulimit -a" shows the following
socket buffer size (bytes, -b) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 524288
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 11095 <-------------- Samee as
kern.maxfilesperproc
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 65536
cpu time (seconds, -t) unlimited
max user processes (-u) 5547
virtual memory (kbytes, -v) unlimited

"sysctl -a | grep kern.maxfile" gives the following kernel limits:
kern.maxfiles: 12328
kern.maxfilesperproc: 11095

PostGreSQL seems to have settled around 4800 open file descriptors as
counted by "lsof | grep postgres | wc -l" and the number of open files
around 3500 as indicated by "sysctl -a | grep kern.openfile"

The only things that's running on the machine is PostGreSQL, hence first
suspicion naturally falls on the database.
It just seems odd that it'd start behaving like this after having run with
no problems for 6 months.

Any insight or suggestions as to where to start digging would be greatly
appreciated

Cheers
Jona

On Wed, Aug 12, 2009 at 7:52 PM, jinson abraham <abraham(dot)jinson(at)gmail(dot)com>wrote:

> since the message is "*out of file descriptors: Too many open files in
> system; release and retry*" i think you should also be checking the system
> ulimit.
>
> try commands like "ulimit -a" to check the number of files currently open
> in the system. Also try "ulimit -Hn" to get "The maximum number of open
> file descriptors (most systems do not allow this value to be set)" see man
> pages of ulimit for more details.
>
> you could also increase the ulimit by making changes in
> "ac/etc/security/limits.conf".
>
> However i hope you have already checked the possibility of some other
> process which might be continuously opening files and not closing it
> properly. Linux has a fixed number of fd's that can be used and if the
> process dont relese them it coudl lead to such a problem.
>
> Hope this helps.
>
> Thanks,
> - Jinson.
>
>
> On Wed, Aug 12, 2009 at 10:44 PM, Emanuel Calvo Franco <
> postgres(dot)arg(at)gmail(dot)com> wrote:
>
>> 2009/8/12 Jonatan Evald Buus <jonatan(dot)buus(at)cellpointmobile(dot)com>:
>> > Greetings,
>> > We're seeing numerous "LOG: out of file descriptors: Too many open
>> files in
>> > system; release and retry" entries as well as quite a few "LOG: could
>> not
>> > open temporary statistics file "global/pgstat.tmp": Too many open files
>> in
>> > system"
>> > Much more alarming however, we're seeing errors such as:
>> > FATAL: could not open file "global/pg_database": Too many open files in
>> > system
>> > ERROR: could not open relation 1663/2219053/2601: Too many open files
>> in
>> > system
>> >
>> > kern.maxfiles is currently set to 12328 and "sysctl -a | grep
>> > kern.openfiles" showed over 10.000 open files prior to a reboot.
>> > After the reboot kern.openfiles quickly increased from less than 200 to
>> over
>>
>> Did you request about kern.maxfiles?
>>
>>
>> --
>> Emanuel Calvo Franco
>> Database consultant at:
>> www.siu.edu.ar
>> www.emanuelcalvofranco.com.ar
>>
>> --
>> Sent via pgsql-admin mailing list (pgsql-admin(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-admin
>>
>
>

--
JONATAN EVALD BUUS

Executive Vice President Open Systems and Telecommunications

Mobile US +1 (305) 331-5242
Mobile DK +45 2888 2861
Telephone +1 (305) 777-0392
Fax. +1 (305) 777-0449
jonatan(dot)buus(at)cellpointmobile(dot)com
www.cellpointmobile.com

CellPoint Mobile Inc.
4000 Ponce de Leon Boulevard
Suite 470
Coral Gables, FL 33146
USA

'Mobilizing the Enterprise'