Re: Huge backend memory footprint

On 22.12.2017 16:21, Andres Freund wrote:
> Hi,
>
> On 2017-12-22 16:07:23 +0300, Konstantin Knizhnik wrote:
>> While my experiments with pthreads version of Postgres I find out that I can
>> not create more than 100k backends even at the system with 4Tb of RAM.
> I don't think this is a problem we need to address at this point. Would
> you care to argue otherwise?

I am not sure that be able to support 100k sessions is really the most
topical goal.
Some modern systems has hundreds of cores and to be able to utilize them
we need thousands of backends.
But not yet 100k backends.

If Postgres can efficiently support arbitrary number of backends
(actually managing more than 100k network connections is problematic in
any case)
then we do not need to use connection pooling. And it definitely has
some clear advantages: use of session variables, prepared statements,...
Also any intermediate proxy can only decrease speed.

But certainly there are a lot of problems except this few arrays which
cause quadratic increase of backends memory footprint with increasing
max_connections.
Postgres backens are heavy wieght: each of them maintains is own private
relation and catalog caches, prepared statement cache,...
I expect that for real large databases them will consume most of the
backend's memory.
So may be the right apporach is to replace this private caches with
shared caches.

>
> For now we've so much more relevant limitations, e.g. the O(N)
> complexity of computing a snapshot, that I think addressing these
> concerns is a waste of time at this point.
Well, we have CSN patch which now is stable enough and shows reaults
comparable with mainstream Postgres. So may be snapshot calculation is
not the most critical problem...

>
> Greetings,
>
> Andres Freund
>

--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company