Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug

On Wed, Aug 20, 2025 at 10:45:13AM -0700, Adrian Klaver wrote:
> On 8/20/25 09:08, hubert depesz lubaczewski wrote:
> > On Wed, Aug 20, 2025 at 08:14:47AM -0700, Adrian Klaver wrote:
> > > Hmm.
> > >
> > > From initial post:
> > >
> > > "For ~ 1 second there are no logs going to log (we usually have at 5-20
> > > messages logged per second), no connection, nothing. And then we get
> > > bunch (30+) messages with the same milisecond time."
> > > Are the 30+ messages all coming in on one connection or multiple
> > > connections?
> > Multiple connections.
> > > Also to be clear these are statements that are being run on the replica
> > > locally, correct?
> > What do you mean locally?
> I should have been clearer. Are the queries being run against the replica or
> the primary?

All to replica. Primary has its own work, of course, but the problem
we're experiencing is on replicas.

> > Application servers are *not* on the same server as the db. So no, they
> > are not local.
> Where are the application servers relative to the replica server?

Well, there is a lot of them. All are in AWS EC2, same region, various
availability zones. Generally we use 3 AZs, with more or less equal
split, so I'd say ~ 33% of app servers is within the same AZ.

> How many applications servers are hitting the database?

To be honest, I'm not sure. I have visibility into dbs, and bouncers,
not really into Apps. I know that these are automatically dynamically
scaled, so number of app server is very varying.

I'd say anything from 40 to 200 app servers hit first layer of bouncers,
which we usually have 6-9 (2-3 per az).

These go to 2nd layer of bouncers, on the db server itself.

Best regards,

depesz