Re: Automatically setting work_mem

Martijn van Oosterhout <kleptog(at)svana(dot)org> writes:
> There is one way to guarentee the memory is released to the OS after
> completion. Make the allocator allocate work_mem bytes using mmap()
> rather than malloc(). munmap() will then definitly return the memory to
> the OS. Unfortunatly, the coding required would probably not be
> straight-forward...

Nor portable.

> Glibc will only convert malloc() to an mmap() on
> allocations > 128KB and I don't think PostgreSQL ever does that.

Actually, we do: it doesn't take very long for the sequence of block
allocations within a context to ramp up to 128K. (And I wouldn't be
opposed to tweaking the logic in aset.c to make it happen faster, once
an initial small allocation is filled up.) Also, individual chunk
requests exceeding 8K or thereabouts are fed directly to malloc, so
stuff like the SortTuple array might well be effectively mmap'd.

I'm fairly unconvinced about Simon's underlying premise --- that we
can't make good use of work_mem in sorting after the run building phase
--- anyway. If we cut back our memory usage then we'll be forcing a
significantly more-random access pattern to the temp file(s) during
merging, because we won't be able to pre-read as much at a time.

regards, tom lane