Re: Add tracking of backend memory allocated to pg_stat_activity

From: John Morris <john(dot)morris(at)crunchydata(dot)com>
To: "reid(dot)thompson(at)crunchydata(dot)com" <reid(dot)thompson(at)crunchydata(dot)com>, Justin Pryzby <pryzby(at)telsasoft(dot)com>
Cc: vignesh C <vignesh21(at)gmail(dot)com>, Andres Freund <andres(at)anarazel(dot)de>, "pgsql-hackers(at)lists(dot)postgresql(dot)org" <pgsql-hackers(at)lists(dot)postgresql(dot)org>, "Drouvot, Bertrand" <bertranddrouvot(dot)pg(at)gmail(dot)com>, Melanie Plageman <melanieplageman(at)gmail(dot)com>, Kyotaro Horiguchi <horikyota(dot)ntt(at)gmail(dot)com>, StephenFrost <sfrost(at)snowman(dot)net>, Ted Yu <yuzhihong(at)gmail(dot)com>
Subject: Re: Add tracking of backend memory allocated to pg_stat_activity
Date: 2023-08-31 16:18:57
Message-ID: BYAPR13MB267710BCC29F13F8BE97E643A0E5A@BYAPR13MB2677.namprd13.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Here is an updated version of the earlier work.

This version:
1) Tracks memory as requested by the backend.
2) Includes allocations made during program startup.
3) Optimizes the "fast path" to only update two local variables.
4) Places a cluster wide limit on total memory allocated.

The cluster wide limit is useful for multi-hosting. One greedy cluster doesn't starve
the other clusters of memory.

Note there isn't a good way to track actual memory used by a cluster.
Ideally, we like to get the working set size of each memory segment along with
the size of the associated kernel data structures.
Gathering that info in a portable way is a "can of worms".
Instead, we're managing memory as requested by the application.
While not identical, the two approaches are strongly correlated.

The memory model used is
1) Each process is assumed to use a certain amount of memory
simply by existing.
2) All pg memory allocations are counted, including those before
the process is fully initialized.
3) Each process maintains its own local counters. These are the "truth".
4) Periodically,
- local counters are added into the global, shared memory counters.
- pgstats is updated
- total memory is checked.

For efficiency, the global total is an approximation, not a precise number.
It can be off by as much as 1 MB per process. Memory limiting
doesn't need precision, just a consistent and reasonable approximation.

Repeating the earlier benchmark test, there is no measurable loss of performance.

Attachment Content-Type Size
memtrack-v4.patch application/octet-stream 107.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Jeff Davis 2023-08-31 16:50:45 Re: [17] CREATE SUBSCRIPTION ... SERVER
Previous Message David Christensen 2023-08-31 16:13:18 Re: Initdb-time block size specification