Re: Vacuum: allow usage of more than 1GB of work mem

On 07/16/2018 10:34 AM, Claudio Freire wrote:
> On Fri, Jul 13, 2018 at 5:43 PM Andrew Dunstan
> <andrew(dot)dunstan(at)2ndquadrant(dot)com> wrote:
>>
>>
>> On 07/13/2018 09:44 AM, Heikki Linnakangas wrote:
>>> On 13/07/18 01:39, Andrew Dunstan wrote:
>>>> On 07/12/2018 06:34 PM, Alvaro Herrera wrote:
>>>>> On 2018-Jul-12, Andrew Dunstan wrote:
>>>>>
>>>>>> I fully understand. I think this needs to go back to "Waiting on
>>>>>> Author".
>>>>> Why? Heikki's patch applies fine and passes the regression tests.
>>>> Well, I understood Claudio was going to do some more work (see
>>>> upthread).
>>> Claudio raised a good point, that doing small pallocs leads to
>>> fragmentation, and in particular, it might mean that we can't give
>>> back the memory to the OS. The default glibc malloc() implementation
>>> has a threshold of 4 or 32 MB or something like that - allocations
>>> larger than the threshold are mmap()'d, and can always be returned to
>>> the OS. I think a simple solution to that is to allocate larger
>>> chunks, something like 32-64 MB at a time, and carve out the
>>> allocations for the nodes from those chunks. That's pretty
>>> straightforward, because we don't need to worry about freeing the
>>> nodes in retail. Keep track of the current half-filled chunk, and
>>> allocate a new one when it fills up.
>>
>> Google seems to suggest the default threshold is much lower, like 128K.
>> Still, making larger allocations seems sensible. Are you going to work
>> on that?
> Below a few MB the threshold is dynamic, and if a block bigger than
> 128K but smaller than the higher threshold (32-64MB IIRC) is freed,
> the dynamic threshold is set to the size of the freed block.
>
> See M_MMAP_MAX and M_MMAP_THRESHOLD in the man page for mallopt[1]
>
> So I'd suggest allocating blocks bigger than M_MMAP_MAX.
>
> [1] http://man7.org/linux/man-pages/man3/mallopt.3.html

That page says:

       M_MMAP_MAX
              This parameter specifies the maximum number of allocation
              requests that may be simultaneously serviced using mmap(2).
              This parameter exists because some systems have a limited
              number of internal tables for use by mmap(2), and using more
              than a few of them may degrade performance.

              The default value is 65,536, a value which has no special
              significance and which serves only as a safeguard. Setting
              this parameter to 0 disables the use of mmap(2) for servicing
              large allocation requests.

I'm confused about the relevance.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services