Re: many sessions waiting DataFileRead and extend

From: James Pang <jamespang886(at)gmail(dot)com>
To: Laurenz Albe <laurenz(dot)albe(at)cybertec(dot)at>
Cc: Frits Hoogland <frits(dot)hoogland(at)gmail(dot)com>, pgsql-performance(at)lists(dot)postgresql(dot)org
Subject: Re: many sessions waiting DataFileRead and extend
Date: 2025-06-26 03:48:45
Message-ID: CAHgTRfcZPWiw7zfBuFub59W5OURcZ2q67HsydTzLhU+54uowXQ@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-performance

Thanks, I make a summary of the issue, no connection storm(fork)
either, just suddenly many session waiting on "extend" and "DataFileRead",
it last 2 seconds, this server has 64 vcpu and running there long time
without issue, only last weekend, we patch from 14.8 to 14.14. We checked
with Infra and VMWARE, and OS monitoring, except very high %sys cpu, not
found other abnormal events. so, we are thinking if Postgresql buffer cache
or OS page cache level contention leading to that(if that can contribute to
IO wait, DataFileRead, extend).
attached please find details, just masking dbname, username,and ip
addresses.

Laurenz Albe <laurenz(dot)albe(at)cybertec(dot)at> 於 2025年6月26日週四 上午11:03寫道:

> On Thu, 2025-06-26 at 10:32 +0800, James Pang wrote:
> > thans for you suggestions, we have iowait from sar command too, copy
> here, checking with infra team not found abnormal IO activities either.
> > 02:00:01 PM CPU %usr %nice %sys %iowait %irq %soft %steal
> %guest %gnice %idle
> > 02:00:03 PM all 15.92 0.00 43.02 0.65 0.76 2.56 0.00
> 0.00 0.00 37.09
>
> Crazy high system time.
>
> If it is not transparent hugepages, the other suspect is fork().
> Turn on "log_connections" for a minute or two and see if there are lots
> of connections established. That can easily use your CPU.
>
> Other than that, I am out of guesses.
>
> > Frits Hoogland <frits(dot)hoogland(at)gmail(dot)com> 於 2025年6月25日週三 下午10:27寫道:
> > > > On 25 Jun 2025, at 07:59, Laurenz Albe <laurenz(dot)albe(at)cybertec(dot)at>
> wrote:
> > > >
> > > > Run "sar -P all 1" and see if "%iowait" is high.
> > >
> > > I would (strongly) advise against the use of iowait as an indicator.
>
> I have heard that before, and I am sure you are right.
>
> I grant that if it is low, it may just mean that the CPU is under load.
> But if %iowait is high, my experience is that that indicates an I/O
> problem.
>
> Yours,
> Laurenz Albe
>

Attachment Content-Type Size
many_sessions_waiting_extend_Datafileread_but_no_ioload_spike application/octet-stream 8.9 KB

In response to

Browse pgsql-performance by date

  From Date Subject
Next Message Motog Plus 2025-06-26 05:11:09 Re: Guidance Needed: Scaling PostgreSQL for 12 TB Data Growth - New Feature Implementation
Previous Message Tatsuo Ishii 2025-06-26 03:25:45 Re: many sessions waiting DataFileRead and extend