Re: VACUUM memory management

On Mon, Dec 9, 2019 at 11:54 PM Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>
wrote:

> On 2019-Dec-09, Ibrar Ahmed wrote:
>
> > Hi,
> >
> > The memory consumption of VACUUM has some issues and could be improved.
> > Some of its limitations are recorded in the comments of the
> “vacuumlazy.c”
> > file. The current design of VACUUM memory usage is that it stores the TID
> > in a fixed-size array which is allocated at the start, based upon
> > maintenance_work_mem. There are three problems with that design
>
> Did you see this thread?
>
> https://postgr.es/m/CAGTBQpbDCaR6vv9=scXzuT8fSbckf=a3NgZdWFWZbdVugVht6Q@mail.gmail.com
>
> Yes, and somehow did what is explained.

Robert: "What I think we need to do is make some provision to initially
allocate only a small amount of memory and then grow the allocation
later if needed. For example, instead of having
vacrelstats->dead_tuples be declared as ItemPointer, declare it as
ItemPointer * and allocate the array progressively in segments. I'd
actually argue that the segment size should be substantially smaller
than 1 GB, like say 64MB; there are still some people running systems
which are small enough that allocating 1 GB when we may need only 6
bytes can drive the system into OOM."

I change vacrelstats->dead_tuples to ItemPointer * and allocate small
memory and added more when needed. What I did new is
divide maintenance_work_mem in fixed-size chunks.

> --
> Álvaro Herrera https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>

--
Ibrar Ahmed