Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug

On 8/21/25 03:07, hubert depesz lubaczewski wrote:
> On Wed, Aug 20, 2025 at 10:45:13AM -0700, Adrian Klaver wrote:
>> On 8/20/25 09:08, hubert depesz lubaczewski wrote:
>>> On Wed, Aug 20, 2025 at 08:14:47AM -0700, Adrian Klaver wrote:
>>>> Hmm.
>>>>
>>>> From initial post:
>>>>
>>>> "For ~ 1 second there are no logs going to log (we usually have at 5-20
>>>> messages logged per second), no connection, nothing. And then we get
>>>> bunch (30+) messages with the same milisecond time."
>>>> Are the 30+ messages all coming in on one connection or multiple
>>>> connections?
>>> Multiple connections.
>>>> Also to be clear these are statements that are being run on the replica
>>>> locally, correct?
>>> What do you mean locally?
>> I should have been clearer. Are the queries being run against the replica or
>> the primary?
>
> All to replica. Primary has its own work, of course, but the problem
> we're experiencing is on replicas.

If I am following there is more then one primary --> replica pair and
the problem exists across all the pairs.

>> How many applications servers are hitting the database?
>
> To be honest, I'm not sure. I have visibility into dbs, and bouncers,
> not really into Apps. I know that these are automatically dynamically
> scaled, so number of app server is very varying.
>
> I'd say anything from 40 to 200 app servers hit first layer of bouncers,
> which we usually have 6-9 (2-3 per az).
>
> These go to 2nd layer of bouncers, on the db server itself.

By bouncer I assume you mean something like pgBouncer, a connection pooler.

Is it possible to determine what bouncer the queries in question are
coming from?

>
> Best regards,
>
> depesz
>

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com