Re: Init connection time grows quadratically

On Fri, 11 Sept 2026, 21:51 Andres Freund, <andres(at)anarazel(dot)de> wrote:
>
> Hi,
>
> On 2026-09-09 15:04:53 +0200, Matthias van de Meent wrote:
> > From a62922a732b261ccfe5c7623aa4fe0e9b18c4aeb Mon Sep 17 00:00:00 2001
> > From: Maksim Melnikov <m(dot)melnikov(at)postgrespro(dot)ru>
> > Date: Tue, 18 Nov 2025 17:20:09 +0300
> > Subject: [PATCH v5] This patch reduce connection init/close time.
> >
> > ProcArrayRemove/ProcArrayAdd are expensive in terms of accessing pgxactoff
> > field in PGPROC, because its are placed on different pages and it the
> > reason of page_faults occurence. Now one of the use case with pgxactoff
> > is iteration with gaps over PGPROCs and read/write pgxactoff, PGPROC
> > allocate ~1KB and it quite enough to have page_faults in case of such
> > accessing.
> >
> > So we placed all pgxactoff in the separate array pgxactoffs to have adjacent
> > pages for them all. Indexes in allProcs aligned with pgxactoffs array, so
> > the Nth element in pgxactoffs refer to Nth PGPROC.
> >
> > Eventually it helps to avoid extra page_faults and reduce connection
> > init/close time. The main benefit is seen on configurations without huge
> > pages.
>
> Maybe I am just missing something, but is this ever a benefit outside of
> completely arbitrary scenarios? Who has >= 8k concurrent connections doing
> nothing but connecting/disconnecting? Even if one were to agree that it's
> useful to optimize large huge_pages=off workloads, surely that's an absurd
> enough workload that nobody cares?

Prepared transactions are also a workload that go through
ProcArrayAdd+Remove, and I don't think that such a workload (with 100s
or even 1000s of prepared xacts) is completely absurd. Uncommon,
sure, but unlike systems with multiple 1000s of connections I would
not call it absurd.

> This isn't entirely free, needing to look at a separate cachelines that are
> frequently modified (and thus commonly won't be in the cpu-local cache), in
> reasonably common codepaths like ProcArrayEndTransactionInternal(),
> TransactionIdIsInProgress(), GetSnapshotData() is far from free.

This is true, but if it is likely that the new pgxactoff cache line is
modified, then it's more likely than not that on HEAD our Proc entry's
xmin/xid cache line would've been modified instead (depending on cache
line size etc). If we're already going to look at nonlocally dirtied
cache lines, why not make sure those lines are densely packed with
dirtied information, so that the modifying backend can be done with it
sooner, and with fewer false sharing opportunities? Note that
pgxactoff is only dirtied if a proc with procnumber smaller than ours
is added or removed, or (with patch) smaller than the last procnumber
on the cacheline.

Currently, ProcArrayAdd/Remove in HEAD frequently falsely shares a
proc's own xmin/xmax, and I don't believe that that dependency is
free, either. Yes, the offsets array's cache line is more likely to
be modified because more backends' offsets fit on the cache line, and
so you trade false sharing your proc's xmin/xmax with other procs'
offsets, but I think that the overall cost is lower, because fewer
recently modified cache lines total need to be modified and moved
around the NUMA domains -- especially given that these are now only
modified by ProcArrayAdd/Remove's code, when we're already under very
heavy locks.

Kind regards,

Matthias van de Meent