Re: pg_stat_io_histogram

On Wed, Mar 18, 2026 at 2:29 PM Jakub Wartak
<jakub(dot)wartak(at)enterprisedb(dot)com> wrote:
>
> On Tue, Mar 17, 2026 at 3:17 PM Andres Freund <andres(at)anarazel(dot)de> wrote:
> > On 2026-03-17 13:13:59 +0100, Jakub Wartak wrote:
> > > 1. Concerns about memory use. With v7 I had couple of ideas, and with those
> > > the memory use is really minimized as long as the code is still simple
> > > (so nothing fancy, just some ideas to trim stuff and dynamically allocate
> > > memory). I hope those reduce memory footprint to acceptable levels, see my
> > > earlier description for v7.
> >
> > Personally I unfortunately continue to think that storing lots of values that
> > are never anything but zero isn't a good idea once you have more than a
> > handful of kB. Storing pointless data is something different than increasing
> > memory usage with actual information.
> >
> > I still think you should just count the number of histograms needed, have an
> > array [object][context][op] with the associated histogram "offset" and then
> > increment the associated offset. It'll add an indirection at count time, but
> > no additional branches.
>
> Great idea, thanks, I haven't thought about that! Attached v9 attempts to do
> that for pending backend I/O struct, which minimizes the (backend) memory
> footprint for client backends to just about ~5kB.
>
> I have been pulling my hair trying to achieve the same for shared-memory, but I
> have failed to do that w/o sinking into complexity [..]

OK, I've made it done too with indirect offset on shared memory, it wasn't easy
at least for me, but now we have two approaches/patchsets:

v9a: as previously with easier code condensing shm memory (for some
backends that
are not used), easier code

v9b: with more code and build complexity but that should address concern of not
used memory

'Shared Memory Stats' allocated size:
master - uses ~308kB for shm
v9a-000[12]: 578kB shm
v9a-000[123]: 507kB shm
v9a-000[1234]: 471kB shm (+~163kB more)

v9b-000[123]: 361kB shm

v9a-000[12] are identical to v9b-00[12], but included just for
patchset completeness.

In v9b meson/autoconf (for adding pgstat_io_genstats) build most of
the time what
they need, but probably that needs some cleanups and better dependency
tracking. I'm
not sure about correctnes of those changes as especially
autoconf/Makefile is a lot
like brainf**k to me and that area would need some help...

I think now we could even increase max resolution of buckets to cover
max those maximum
of 32s+ (at the cost of one extra 64-byte cacheline for pending IO
stats, so go with
PGSTAT_IO_HIST_BUCKETS from 16 to 24)

-J.