Redesign per-backend statistics

Hi hackers,

The current per-backend statistics design excludes several process types and
duplicates accounting and storage: WAL, lock, and IO activity is reported both to
the global statistics and to PGSTAT_KIND_BACKEND.

This duplicated work was raised as a concern in [1], and its cost would grow as
more statistics acquire per-backend reporting.

Andres proposed a new design in [2]:

"
I think what we'd want is rather something where for each applicable stats kind
we have a shared counter for all exited backends and then per-backend counters
for live backends, with helpers to aggregate the exited + live stats to a total
"

and this is what the attached patch series is implementing.

Its main benefits are:

1/ each counter is reported and stored once, rather than in both the fixed global
statistics and PGSTAT_KIND_BACKEND.

2/ WAL, lock, and IO flushes update a per backend entry without acquiring
the corresponding global statistics lock or performing a hash lookup. This removes
global lock contention from the flush path.

3/ WAL, lock, and IO statistics have independent storage and locking instead of
sharing one PGSTAT_KIND_BACKEND entry.

4/ per-backend statistics become available for auxiliary processes previously
excluded by the backend type filtering.

There are 2 tradeoff though:

1/ fetching global statistics now requires combining all live entries.

2/ a shared statistics reset now clears both the global stats and all live entries.
Otherwise, values from live entries would immediately reappear in the global view.

The first tradeoff moves work from the frequent flush path to the comparatively
infrequent query path. The second is a documented behavior change.

The patch series is organized that way:

0001: add tests for per-backend statistics

It adds new tests that will serve as compatibility coverage for the redesign.
It could be applied while we are discussing the other patches.

0002: add new infrastructure for per-backend statistics

It introduces the common per-backend entry header, the kind metadata needed to
describe per-backend storage, and backend-local state holding the attached dshash
and cached entry pointer. It provides the common operations for creating and
attaching the hashes, creating and caching the current process's entries, fetching
an individual entry, constructing consistent snapshots, transferring entries into
global statistics, removing entries, and the accumulation at clean server shutdown.

No statistics kind registers per-backend metadata in this patch.

0003 moves WAL statistics to the infrastructure introduced by 0002.
0004 performs the corresponding conversion for lock statistics.
0005 performs the corresponding conversion for IO statistics. Once WAL, lock,
and IO have moved, PGSTAT_KIND_BACKEND contains no data, so this patch also removes
that kind and pgstat_backend.c, together with their obsolete infrastructure.

Design explanation for the new hashes:

- a fixed array indexed by ProcNumber would avoid hash operations, but it would
reserve shared memory for every possible process slot and every participating kind.
Queries could also have to inspect unused slots. The dshash allocates entries
for processes that actually exist and lets queries iterate those entries directly.

- reusing the current variable numbered statistics hash would require aggregate
queries to scan unrelated statistics entries or require another structure for
enumerating only the live entries of that kind. A dedicated per kind hash provides
that enumeration directly.

Remark:

The patch series limits this new infrastructure to built in fixed numbered
statistics kinds as this is the only use case we have had so far. We could extend
to variable ones later on if needed.

[1]: https://postgr.es/m/7fhpds4xqk6bnudzmzkqi33pinsxammpljwde5gfkjdygvejrj@ojkzfr7dxkmm
[2]: https://postgr.es/m/et272fdhdx6yphlgzvrgsf7bgwnf3vqciwp4gxqubro42uaflp%40ohslaocvwgvi

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com