Re: Estimating HugePages Requirements?

At Thu, 2 Sep 2021 16:46:56 +0000, "Bossart, Nathan" <bossartn(at)amazon(dot)com> wrote in
> On 9/2/21, 12:54 AM, "Michael Paquier" <michael(at)paquier(dot)xyz> wrote:
> > Thanks. Anyway, we don't really need huge_pages_required on Windows,
> > do we? The following docs of Windows tell what do to when using large
> > pages:
> > https://docs.microsoft.com/en-us/windows/win32/memory/large-page-support
> >
> > The backend code does that as in PGSharedMemoryCreate(), now that I
> > look at it. And there is no way to change the minimum large page size
> > there as far as I can see because that's decided by the processor, no?
> > There is a case for shared_memory_size on Windows to be able to adjust
> > the sizing of the memory of the host, though.
>
> Yeah, huge_pages_required might not serve much purpose for Windows.
> We could always set it to -1 for Windows if it seems like it'll do
> more harm than good.

I agreed to this.

> > At the end it would be nice to not finish with two GUCs. Both depend
> > on the reordering of the actions done by the postmaster, so I'd be
> > curious to hear the thoughts of others on this particular point.
>
> Of course. It'd be great to hear others' thoughts on this stuff.

Honestly, I would be satisfied if the following error message
contained required huge pages.

FATAL: could not map anonymous shared memory: Cannot allocate memory
HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 148897792 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.

Or emit a different message if huge_pages=on.

FATAL: could not map anonymous shared memory from huge pages
HINT: This usually means that PostgreSQL's request for huge pages more than available. The required 2048kB huge pages for the required memory size (currently 148897792 bytes) is 71 pages.

Returning to this feature, even if I am informed that via GUC, I won't
add memory by looking shared_memory_size. Anyway since shard_buffers
occupies almost all portion of shared memory allocated to postgres, we
are not supposed to need such a precise adjustment of the required
size of shared memory. On the other hand available number of huge
pages is configurable and we need to set it as required. On the other
hand, it might seem to me a bit strange that there's only
huge_page_required and not shared_memory_size in the view of
comprehensiveness or completeness. So my feeling at this point is "I
need only huge_pages_required but might want shared_memory_size just
for completeness".

By the way I noticed that postgres -C huge_page_size shows 0, which I
think should have the number used for the calculation if we show
huge_page_required.

I noticed that postgres -C shared_memory_size showed 137 (= 144703488)
whereas the error message above showed 148897792 bytes (142MB). So it
seems that something is forgotten while calculating
shared_memory_size. As the consequence, launching postgres setting
huge_pages_required (69 pages) as vm.nr_hugepages ended up in the
"could not map anonymous shared memory" error.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center