Re: Add tracking of backend memory allocated to pg_stat_activity

On Mon, 2022-11-28 at 10:59 -0800, Andres Freund wrote:
> On 2022-11-26 22:10:06 -0500, Reid Thompson wrote:
> >    - zero allocated bytes after fork to avoid double counting
> > postmaster allocations
>
> I still don't understand this - postmaster shouldn't be counted at
> all. It
> doesn't have a PgBackendStatus. There simply shouldn't be any tracked
> allocations from it.
>
> Greetings,
>
> Andres Freund

Hi Andres,
I based this on the following.

It appears to me that Postmaster populates the local variable that
*my_allocated_bytes points to. That allocation is passed to forked
children, and if not zeroed out, will be double counted as part of
the child allocation. Is this invalid?

$ ps -ef|grep postgres
postgres 6389 1 0 Dec01 ? 00:00:17 /usr/sbin/pgbouncer -d /etc/pgbouncer/pgbouncer.ini
rthompso 2937799 1 0 09:45 ? 00:00:00 /tmp/postgres/install/pg-stats-memory/bin/postgres -D /var/tmp/pg-stats-memory -p 5433
rthompso 2937812 2937799 0 09:45 ? 00:00:00 postgres: checkpointer
rthompso 2937813 2937799 0 09:45 ? 00:00:00 postgres: background writer
rthompso 2937816 2937799 0 09:45 ? 00:00:00 postgres: walwriter
rthompso 2937817 2937799 0 09:45 ? 00:00:00 postgres: autovacuum launcher
rthompso 2937818 2937799 0 09:45 ? 00:00:00 postgres: logical replication launcher
rthompso 2938877 2636586 0 09:46 pts/4 00:00:00 /usr/lib/postgresql/12/bin/psql -h localhost -p 5433 postgres
rthompso 2938909 2937799 0 09:46 ? 00:00:00 postgres: rthompso postgres 127.0.0.1(44532) idle
rthompso 2942164 1987403 0 09:49 pts/3 00:00:00 grep postgres

Bracketing fork_process() calls with logging to print *my_allocated_bytes immediately prior and after fork_process...
To me, this indicates that the forked children for this invocation
(checkpointer, walwriter, autovac launcher, client backend, autovac worker, etc)
are inheriting 240672 bytes from postmaster.

$ ccat logfile
2022-12-02 09:45:05.871 EST [2937799] LOG: starting PostgreSQL 16devel on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, 64-bit
2022-12-02 09:45:05.872 EST [2937799] LOG: listening on IPv4 address "127.0.0.1", port 5433
2022-12-02 09:45:05.874 EST [2937799] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"
parent StartChildProcess process. pid: 2937799 *my_allocated_bytes: 240672
parent StartChildProcess process. pid: 2937799 *my_allocated_bytes: 240672
parent StartChildProcess process. pid: 2937799 *my_allocated_bytes: 240672
child StartChildProcess process. pid: 2937812 *my_allocated_bytes: 240672
child StartChildProcess process. pid: 2937813 *my_allocated_bytes: 240672
child StartChildProcess process. pid: 2937814 *my_allocated_bytes: 240672
2022-12-02 09:45:05.884 EST [2937814] LOG: database system was shut down at 2022-12-02 09:41:13 EST
parent StartChildProcess process. pid: 2937799 *my_allocated_bytes: 240672
parent StartAutoVacLauncher process. pid: 2937799 *my_allocated_bytes: 240672
child StartChildProcess process. pid: 2937816 *my_allocated_bytes: 240672
parent do_start_bgworker process. pid: 2937799 *my_allocated_bytes: 240672
child StartAutoVacLauncher process. pid: 2937817 *my_allocated_bytes: 240672
2022-12-02 09:45:05.889 EST [2937799] LOG: database system is ready to accept connections
child do_start_bgworker process. pid: 2937818 *my_allocated_bytes: 240672
parent StartAutoVacWorker process. pid: 2937799 *my_allocated_bytes: 240672
child StartAutoVacWorker process. pid: 2938417 *my_allocated_bytes: 240672
parent BackendStartup process. pid: 2937799 *my_allocated_bytes: 240672
child BackendStartup process. pid: 2938909 *my_allocated_bytes: 240672
parent StartAutoVacWorker process. pid: 2937799 *my_allocated_bytes: 240672
child StartAutoVacWorker process. pid: 2938910 *my_allocated_bytes: 240672
parent StartAutoVacWorker process. pid: 2937799 *my_allocated_bytes: 240672
child StartAutoVacWorker process. pid: 2939340 *my_allocated_bytes: 240672
parent StartAutoVacWorker process. pid: 2937799 *my_allocated_bytes: 240672
child StartAutoVacWorker process. pid: 2939665 *my_allocated_bytes: 240672
parent StartAutoVacWorker process. pid: 2937799 *my_allocated_bytes: 240672
child StartAutoVacWorker process. pid: 2940038 *my_allocated_bytes: 240672
parent StartAutoVacWorker process. pid: 2937799 *my_allocated_bytes: 240672
child StartAutoVacWorker process. pid: 2940364 *my_allocated_bytes: 240672
parent StartAutoVacWorker process. pid: 2937799 *my_allocated_bytes: 240672
child StartAutoVacWorker process. pid: 2940698 *my_allocated_bytes: 240672
parent StartAutoVacWorker process. pid: 2937799 *my_allocated_bytes: 240672
child StartAutoVacWorker process. pid: 2941317 *my_allocated_bytes: 240672
parent StartAutoVacWorker process. pid: 2937799 *my_allocated_bytes: 240672
child StartAutoVacWorker process. pid: 2941825 *my_allocated_bytes: 240672

Reid Thompson
Senior Software Engineer
Crunchy Data, Inc.

reid(dot)thompson(at)crunchydata(dot)com
www.crunchydata.com