Limiting memory allocation

Greetings,

An ongoing issue in container environments where Kubernetes is being
used is that setting the overcommit parameters on the base system will
impact all of the processes on that system and not all of them handle
malloc failing as gracefully as PG does and may allocate more than what
they really need with the expectation that it'll work. Folks have
already started hacking around this issue by using things like
LD_PRELOAD'd libraries to put code between palloc and malloc, but that
could cause any malloc to fail and it strikes me that, in an ideal
world, we could give users a way to constrain the total amount of memory
allocated by regular backends (or those where a user is able to control
how much memory is allocated, more generally) while not impacting the
processes which keep the server running.

I wanted to bring this general idea up for discussion here to get
feedback on the concept before going off to write code for it. Seems
unlikely that it would be a huge amount of code while there's likely to
need to be discussion about how one would configure this and how we
might handle parallel query and such.

Empirically, the LD_PRELOAD hack does, in fact, seem to work, so even a
simple approach would be helpful, but certainly there is an angle to
this where we might eventually allow certain backends (perhaps certain
roles, etc) to allocate more and others to not be allowed to allocate as
much, etc. Ideally we would look to go with the simple approch first
and then we can contemplate making it more complicated in the future and
not try to accomplish it all in the first pass. Of course, we should
keep in mind such ideas and try to avoid anything which would preclude
us for adding that flexibility in the future.

Thoughts?

Thanks,

Stephen