Re: Estimating HugePages Requirements?

From: "Bossart, Nathan" <bossartn(at)amazon(dot)com>
To: Don Seiler <don(at)seiler(dot)us>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Estimating HugePages Requirements?
Date: 2021-06-09 20:52:47
Message-ID: F2772387-CE0F-46BF-B5F1-CC55516EB885@amazon.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin pgsql-hackers

moving to pgsql-hackers@

On 6/9/21, 9:41 AM, "Don Seiler" <don(at)seiler(dot)us> wrote:
> I'm trying to set up a chef recipe to reserve enough HugePages on a
> linux system for our PG servers. A given VM will only host one PG
> cluster and that will be the only thing on that host that uses
> HugePages. Blogs that I've seen suggest that it would be as simple
> as taking the shared_buffers setting and dividing that by 2MB (huge
> page size), however I found that I needed some more.
>
> In my test case, shared_buffers is set to 4003MB (calculated by
> chef) but PG failed to start until I reserved a few hundred more MB.
> When I checked VmPeak, it was 4321MB, so I ended up having to
> reserve over 2161 huge pages, over a hundred more than I had
> originally thought.
>
> I'm told other factors contribute to this additional memory
> requirement, such as max_connections, wal_buffers, etc. I'm
> wondering if anyone has been able to come up with a reliable method
> for determining the HugePages requirements for a PG cluster based on
> the GUC values (that would be known at deployment time).

In RDS, we've added a pg_ctl option that returns the amount of shared
memory required. Basically, we start up postmaster just enough to get
an accurate value from CreateSharedMemoryAndSemaphores() and then shut
down. The patch is quite battle-tested at this point (we first
started using it in 2017, and we've been enabling huge pages by
default since v10). I'd be happy to clean it up and submit it for
discussion in pgsql-hackers@ if there is interest.

Nathan

In response to

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Mark Dilger 2021-06-09 22:50:52 Re: Estimating HugePages Requirements?
Previous Message Magnus Hagander 2021-06-09 19:30:52 Re: Estimating HugePages Requirements?

Browse pgsql-hackers by date

  From Date Subject
Next Message Alvaro Herrera 2021-06-09 20:53:09 Re: pg_stat_progress_create_index vs. parallel index builds
Previous Message Mark Zellers 2021-06-09 20:51:54 Re: [External Sender] Re: A modest proposal vis hierarchical queries: MINUS in the column list