Re: Re: Too many open files (was Re: spinlock problems reported earlier)

On Mon, 28 Aug 2000, Tom Lane wrote:

> Brook Milligan <brook(at)biology(dot)nmsu(dot)edu> writes:
> > In any case, if this really follows the POSIX standard, perhaps
> > PostgreSQL code should assume these semantics and work around other
> > cases that don't follow the standard (instead of work around the POSIX
> > cases).
>
> HP asserts that *they* follow the POSIX standard, and in this case
> I'm more inclined to believe them than the *BSD camp. A per-process
> limit on open files has existed in most Unices I've heard of; I had
> never heard of a per-userid limit until yesterday. (And I'm not yet
> convinced that that's actually what *BSD implements; are we sure it's
> not just a typo in the man page?)
>
> 64 or so for _SC_OPEN_MAX is not really what I'm worried about anyway.
> IIRC, we've heard reports that some platforms return values in the
> thousands, ie, essentially telling each process it can have the whole
> kernel FD table, and it's that behavior that I'm speculating is causing
> Marc's problem.
>
> Marc, could you check what is returned by sysconf(_SC_OPEN_MAX) on your
> box? And/or check to see how many files each backend is actually
> holding open?

> ./t
4136

> sysctl kern.maxfiles
kern.maxfiles: 32768

> cat t.c
#include <stdio.h>
#include <unistd.h>

main()
{
printf("%ld\n", sysconf(_SC_OPEN_MAX));
}

okay, slightly difficult since they come and go, but using the database
that is used for the search engine, with just a psql session:

pgsql# lsof -p 85333
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 85333 pgsql cwd VDIR 13,131088 3072 7936 /pgsql/data2/udmsearch
postgres 85333 pgsql rtd VDIR 13,131072 512 2 /
postgres 85333 pgsql txt VREG 13,131084 4651486 103175 /pgsql/bin/postgres
postgres 85333 pgsql txt VREG 13,131076 77648 212924 /usr/libexec/ld-elf.so.1
postgres 85333 pgsql txt VREG 13,131076 11860 56504 /usr/lib/libdescrypt.so.2
postgres 85333 pgsql txt VREG 13,131076 120736 56525 /usr/lib/libm.so.2
postgres 85333 pgsql txt VREG 13,131076 34336 56677 /usr/lib/libutil.so.3
postgres 85333 pgsql txt VREG 13,131076 154128 57068 /usr/lib/libreadline.so.4
postgres 85333 pgsql txt VREG 13,131076 270100 56532 /usr/lib/libncurses.so.5
postgres 85333 pgsql txt VREG 13,131076 570064 56679 /usr/lib/libc.so.4
postgres 85333 pgsql 0r VCHR 2,2 0t0 7967 /dev/null
postgres 85333 pgsql 1w VREG 13,131084 995 762037 /pgsql/logs/postmaster.5432.61308
postgres 85333 pgsql 2w VREG 13,131084 316488878 762038 /pgsql/logs/5432.61308
postgres 85333 pgsql 3r VREG 13,131088 1752 8011 /pgsql/data2/udmsearch/pg_internal.init
postgres 85333 pgsql 4u VREG 13,131084 22757376 15922 /pgsql/data/pg_log
postgres 85333 pgsql 5u unix 0xd46a3300 0t0 ->0xd469a540
postgres 85333 pgsql 6u VREG 13,131084 8192 15874 /pgsql/data/pg_variable
postgres 85333 pgsql 7u VREG 13,131088 16384 7982 /pgsql/data2/udmsearch/pg_class
postgres 85333 pgsql 8u VREG 13,131088 32768 7980 /pgsql/data2/udmsearch/pg_class_relname_index
postgres 85333 pgsql 9u VREG 13,131088 81920 7985 /pgsql/data2/udmsearch/pg_attribute
postgres 85333 pgsql 10u VREG 13,131088 65536 7983 /pgsql/data2/udmsearch/pg_attribute_relid_attnum_index
postgres 85333 pgsql 11u VREG 13,131088 8192 7945 /pgsql/data2/udmsearch/pg_trigger
postgres 85333 pgsql 12u VREG 13,131088 8192 7993 /pgsql/data2/udmsearch/pg_am
postgres 85333 pgsql 13u VREG 13,131088 16384 7977 /pgsql/data2/udmsearch/pg_index
postgres 85333 pgsql 14u VREG 13,131088 8192 7988 /pgsql/data2/udmsearch/pg_amproc
postgres 85333 pgsql 15u VREG 13,131088 16384 7991 /pgsql/data2/udmsearch/pg_amop
postgres 85333 pgsql 16u VREG 13,131088 73728 7961 /pgsql/data2/udmsearch/pg_operator
postgres 85333 pgsql 17u VREG 13,131088 16384 7976 /pgsql/data2/udmsearch/pg_index_indexrelid_index
postgres 85333 pgsql 18u VREG 13,131088 32768 7960 /pgsql/data2/udmsearch/pg_operator_oid_index
postgres 85333 pgsql 19u VREG 13,131088 16384 7976 /pgsql/data2/udmsearch/pg_index_indexrelid_index
postgres 85333 pgsql 20u VREG 13,131088 16384 7942 /pgsql/data2/udmsearch/pg_trigger_tgrelid_index
postgres 85333 pgsql 21u VREG 13,131084 8192 15921 /pgsql/data/pg_shadow
postgres 85333 pgsql 22u VREG 13,131084 8192 15918 /pgsql/data/pg_database
postgres 85333 pgsql 23u VREG 13,131088 8192 7952 /pgsql/data2/udmsearch/pg_rewrite
postgres 85333 pgsql 24u VREG 13,131088 16384 7941 /pgsql/data2/udmsearch/pg_type
postgres 85333 pgsql 25u VREG 13,131088 16384 7940 /pgsql/data2/udmsearch/pg_type_oid_index
postgres 85333 pgsql 26u VREG 13,131088 0 7938 /pgsql/data2/udmsearch/pg_user
postgres 85333 pgsql 27u VREG 13,131088 188416 7984 /pgsql/data2/udmsearch/pg_attribute_relid_attnam_index
postgres 85333 pgsql 28u VREG 13,131088 65536 7959 /pgsql/data2/udmsearch/pg_operator_oprname_l_r_k_index
postgres 85333 pgsql 29u VREG 13,131088 16384 7981 /pgsql/data2/udmsearch/pg_class_oid_index
postgres 85333 pgsql 30u VREG 13,131088 40960 7948 /pgsql/data2/udmsearch/pg_statistic
postgres 85333 pgsql 31u VREG 13,131088 32768 7947 /pgsql/data2/udmsearch/pg_statistic_relid_att_index
postgres 85333 pgsql 32u VREG 13,131088 212992 7958 /pgsql/data2/udmsearch/pg_proc
postgres 85333 pgsql 33u VREG 13,131088 49152 7957 /pgsql/data2/udmsearch/pg_proc_oid_index

when running a vacuum on the database, the only changes appear to be
adding (and removing when done) those tables that are currently being
vacuumed ... so, it appears, ~48 or so files opened ...