Re: Huge backend memory footprint

On 22.12.2017 16:21, Andres Freund wrote:
> Hi,
>
> On 2017-12-22 16:07:23 +0300, Konstantin Knizhnik wrote:
>> While my experiments with pthreads version of Postgres I find out that I can
>> not create more than 100k backends even at the system with 4Tb of RAM.
> I don't think this is a problem we need to address at this point. Would
> you care to argue otherwise?

I am not sure that be able to support 100k sessions is really the most 
topical goal.
Some modern systems has hundreds of cores and to be able to utilize them 
we need thousands of backends.
But not yet 100k backends.

If Postgres can efficiently support arbitrary number of backends 
(actually managing more than 100k network connections is problematic in 
any case)
then we do not need to use connection pooling. And it definitely has 
some clear advantages: use of session variables, prepared statements,...
Also any intermediate proxy can only decrease speed.

But certainly there are a lot of problems except this few arrays which 
cause quadratic increase of backends memory footprint with increasing 
max_connections.
Postgres backens are heavy wieght: each of them maintains is own private 
relation and catalog caches, prepared statement cache,...
I expect that for real large databases them will consume most of the 
backend's memory.
So may be the right apporach is to replace this private caches with 
shared caches.

>
> For now we've so much more relevant limitations, e.g. the O(N)
> complexity of computing a snapshot, that I think addressing these
> concerns is a waste of time at this point.
Well, we have CSN patch which now is stable enough and shows reaults 
comparable with mainstream Postgres. So may be snapshot calculation is 
not the most critical problem...



>
> Greetings,
>
> Andres Freund
>

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company