Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug

On 8/20/25 04:32, hubert depesz lubaczewski wrote:
> On Tue, Aug 19, 2025 at 11:39:03AM -0700, Adrian Klaver wrote:
>>> Every now and then (usually every 3-5 minutes, but not through the whole
>>> day), we see situations where every query suddently takes ~ 1 second.
>> Given the subject line, what you are reporting is happening on the replica,
>> correct?
>
> Yes.
>
>> If so where is the replica relative to the primary, in terms of network
>> distance?
>
> =$ ping -c 10 primary
> reports:
> 10 packets transmitted, 10 received, 0% packet loss, time 9181ms
> rtt min/avg/max/mdev = 0.942/0.956/0.991/0.012 ms
>
>> Also what are the 'hardware' specifications on the replica instance?
>
> c8g.48xlarge ec2 instance. It is arm64, 192 cores, with 384 gb of ram.
>
> As for storage, this is relatitvely slow, because this db is rather
> small:
> gp3 500gb volume, with 6000 iops. At no point is IO in any way close to
> limits, the whole db fits easily in ram.

Hmm.

From initial post:

"For ~ 1 second there are no logs going to log (we usually have at 5-20
messages logged per second), no connection, nothing. And then we get
bunch (30+) messages with the same milisecond time."

Are the 30+ messages all coming in on one connection or multiple
connections?

Also to be clear these are statements that are being run on the replica
locally, correct?

Does the AWS monitoring indicate any issues?

>
> Best regards,
>
> depesz
>

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com