Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

From: surya poondla <suryapoondla4(at)gmail(dot)com>
To: Jelte Fennema-Nio <postgres(at)jeltef(dot)nl>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>, Andres Freund <andres(at)anarazel(dot)de>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, "tomas(at)vondra(dot)me" <tomas(at)vondra(dot)me>, hlinnaka(at)iki(dot)fi, Peter Eisentraut <peter(at)eisentraut(dot)org>, Haibo Yan <tristan(dot)yim(at)gmail(dot)com>
Subject: Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup
Date: 2026-11-17 19:23:37
Message-ID: CAOVWO5rvAkq09+F1yOUsk+8B+ak_mA1ehfAwjGCzjreGrzbqfA@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi Jelte,

Thanks for v12. It addresses the three points from my last mail. I built it
on current master and the basic behavior works, with ulimit -n 256 the
postmaster logs "increased open file limit to 1003", and COPY FROM PROGRAM
sees 256 again in the child.

I see two problems, though:
1. On macOS, startup can loop forever in count_usable_fds(). With the
default unlimited hard limit, ulimit -n 256 and
max_files_per_process=200000 (above kern.maxfilesperproc, 138240 here), the
postmaster never finishes starting and logs "increased open file limit to
..." without end, each step adding 61763. setrlimit() accepts the value,
but dup() still fails with EMFILE at kern.maxfilesperproc. The EMFILE
branch calls IncreaseOpenFileLimit() again, which succeeds again, and we
continue. Master starts fine in the same setup (max_safe_fds = 243).
Perhaps retry on EMFILE only once after each successful increase, i.e.
break if the next dup() still fails, and/or clamp to kern.maxfilesperproc.
2. pg_pclose() returns -1 without calling waitpid() when fclose() fails.
One way to hit this is COPY TO PROGRAM where the program exits without
reading its input. The fwrite() in CopySendEndOfRow() fails with EPIPE, and
the code calls ClosePipeToProgram() to report the exit code. But fclose()
then fails flushing the remaining buffered data, so the exit status is
lost. COPY ... TO PROGRAM 'exit 3' reports on master:

ERROR: program "exit 3" failed
DETAIL: child process exited with exit code 3

and on v12:

ERROR: could not close pipe to external command: Broken pipe
The child is also never reaped, after two such COPYs in one session I see
two <defunct> children of the backend, which remain until the backend exits
(master leaves none). pclose() always waits for the child, so I think
pg_pclose() should always call waitpid() and return the child's status.

A small observation, pipe2 is added to meson.build and pg_config.h.in, but
not to configure.ac, so autoconf builds never define HAVE_PIPE2.

A few smaller things:
1. The IsOpenFileLimit() comment is cut off ("based on our. This should
only be called if").
2. The new comment in 0002 is garbled ("if the result is different than
what the used value turns out to be different than...").
3. SaveOriginalOpenFileLimit() still says "Returns true if successful", and
the rlim_cur + extra_files overflow check from August isn't in yet.
4. In OpenPipeStream(), pqsignal(SIGPIPE, PG_SIG_IGN) after pg_popen()
looks redundant now that the parent no longer sets SIGPIPE to SIG_DFL
before forking.

Regards,
Surya Poondla

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Bharath Rupireddy 2026-11-17 19:24:16 Re: Add tests for concurrent DML retry paths in logical replication apply
Previous Message Kirill Reshke 2026-11-17 18:46:22 Eager aggregation and volatile aggregate support functions