Re: FD_SETSIZE with large #s of files/ports in use

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> "B. Nicholson" <b(dot)nicholson(at)niceng(dot)com> writes:
> > Tom, what libc details will be broken by setting FD_SETSIZE to a larger
> > number? I'm curious for my own knowledge base. I can see where it
> > might cause 'data' sizes to grow which might break thinks. Anything else?
>
> I'm not too sure, honestly. I can tell you that this exact point came up
> recently on a Red Hat internal mailing list, and no less an authority
> than Ulrich Drepper said "you can't do that, it'll break things". He
> didn't say exactly what though. It's possible that on non-glibc-based
> platforms, you could get away with it.

I'd guess that as FD_SETSIZE is a macro used at compile time
(including compile time of libc) and that without jumping
through hoops in the implementation changing it later will
cause inconsistencies between the size of structures or arrays
passed between the application and libc.

At the risk of topic drift and providing more information than
people want to know (but think of the archives! :-), here is
some additional information.

Summary:

a) you can't rely on changing FD_SETSIZE for select(2)
b) poll(2) is preferable to select(2) for performance
c) interfaces that should perform better than either select(2)
and poll(2) are:

i. /dev/poll (Solaris, HP-UX)
ii. epoll (Linux)
iii. pollset (AIX)
iv. kqueue (*BSDs)

There is some hope of maintaining portability across this
newer, non-standardised set of interfaces with libevent.

PostgreSQL seems to use poll() if it's available in some
places, and select() in others. (And I don't know about the
Windows code.)

For small numbers of file descriptors especially on non-hot
code paths, it's not going to matter. In general it would be
IMHO nice to use poll() consistently when it's available
and not emulated via select().

Whether there is a performance gain to be had by using the
non-portable solutions I don't know: it would be interesting
to see some measurements, but I wouldn't necessarily expect
so: the newer interfaces (certainly /dev/poll) were driven by
the needs of high performance web servers with high numbers of
connections which may be a too-different use case to
PostgreSQL to see a notable benefit.

Based on some micro benchmarks I did some years back (on now
non-current OS releases which I shall not name) I would not
assume that the relative performance of these interfaces (that
is, select v. poll v. whatever alternative local enhancement
has been created) would be consistent: you may find systems
with relatively well performing select(2) implementations.

Re point (a):

For POSIX, FD_SETSIZE is not documented as being changeable by
the application, implying that it shouldn't be altered by
portable applications:

http://www.opengroup.org/onlinepubs/009695399/basedefs/sys/select.h.html

That's from POSIX a.k.a. IEEE Std 1003.1, 2004 Edition, a.k.a. the
"Single Unix Specification Version 3".

I'm no Linux guru, so I'll take Tom's and Ulrich Drepper's
word for the behaviour there.

Some operating systems _do_ allow applications to alter
FD_SETSIZE, including at least HP-UX:

http://docs.hp.com/en/B2355-60130/select.2.html

Re point(b):

It is "well known" that poll(2) is more efficient than
select(2) as the sets of file descriptors don't have to be
reset before each call as they are in select(2).

(Sorry, no good reference to hand, and I'm sure someone's had
an exception somewhere, at least when poll() was emulated via
select()!).

Re point(c):

i. /dev/poll was introduced in Solaris 7, and added some time
later to HP-UX:

http://developers.sun.com/solaris/articles/polling_efficient.html
http://docs.hp.com/en/B2355-60130/poll.7.html

ii. Linux preferred to introduce epoll(7):

http://www.kernel.org/doc/man-pages/online/pages/man4/epoll.4.html

iii. IBM preferred pollset for AIX:

http://publib.boulder.ibm.com/infocenter/aix/v6r1/topic/com.ibm.aix.basetechref/doc/basetrf1/pollset.htm

iv. The *BSDs have developed kqueue; originally in FreeBSD but
adopted by NetBSD and OpenBSD:

http://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2
http://netbsd.gw.com/cgi-bin/man-cgi?kqueue++NetBSD-5.0
http://www.openbsd.org/cgi-bin/man.cgi?query=kqueue&sektion=2

v. libevent:

http://www.monkey.org/~provos/libevent/

Regards,

Giles

P.S. No, this isn't the nightmare. I just woke _up_ from the
nightmare. :-) Now, back to sleep ...