Re: Built-in support for a memory consumption ulimit?

On Mon, Jun 16, 2014 at 10:16 PM, Noah Misch <noah(at)leadboat(dot)com> wrote:
> On Sat, Jun 14, 2014 at 10:37:36AM -0400, Tom Lane wrote:
>> After giving somebody advice, for the Nth time, to install a
>> memory-consumption ulimit instead of leaving his database to the tender
>> mercies of the Linux OOM killer, it occurred to me to wonder why we don't
>> provide a built-in feature for that, comparable to the "ulimit -c max"
>> option that already exists in pg_ctl.
>
> In principle, I would like to have such a feature. The only limit I've found
> reliable on the occasions I wanted this was RLIMIT_AS; RLIMIT_DATA has been
> ineffective on Linux+GNU libc. RLIMIT_AS has its own problems, of course:
> address space usage is only tenuously connected to the definition of "memory
> consumption" folks actually want. Worse, one can often construct a query to
> crash an RLIMIT_AS-affected backend. Make the query use heap space until just
> shy of the limit, then burn stack space until RLIMIT_AS halts stack growth.

Ouch. Having a feature that causes excessive memory utilization to
produce an ERROR that halts query execution and returns us to the top
level, as Tom and Greg were proposing, sounds nice, though even there
I wonder what happens if the memory exhaustion is due to things like
relcache bloat which will not be ameliorated by error recovery. But
having a feature that crashes the backend in similar circumstances
doesn't sound nearly as nice.

We could do better by accounting for memory usage ourselves, inside
the memory-context system, but that'd probably impose some overhead we
don't have today.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company