Re: MI - WAL issue

From: sayli khedekar <saylikhedekar1(at)gmail(dot)com>
To: mahamood hussain <hussain(dot)ieg(at)gmail(dot)com>
Cc: Pgsql-admin <pgsql-admin(at)lists(dot)postgresql(dot)org>
Subject: Re: MI - WAL issue
Date: 2026-07-27 11:38:10
Message-ID: CAEUYW+T3EPLAt1Y8qC=tmuipRGzBJEof6sQz9-p_Vv6cE9NTYQ@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-admin

Hi ,

Can't confirm the checkpoint_timeout change directly caused the WAL flush
error, but the 1-min checkpoint interval may add extra write I/O -- two
things worth checking to reduce that load:

1) HOT update ratio + fillfactor — for busiest tables, check
n_tup_hot_upd/n_tup_upd in pg_stat_user_tables. If low (and not due to
indexed columns changing), lower fillfactor to 70–90 to allow more HOT
updates.

2) Unused/duplicate indexes — check pg_stat_user_indexes for near-zero
idx_scan and check for duplicates on the same columns.

pg_wal size may also have increased due to more full page image, but I
don't see its relevance here.

Regards,
Sayli

On Mon, Jul 27, 2026 at 2:32 PM mahamood hussain <hussain(dot)ieg(at)gmail(dot)com>
wrote:

> Thanks for the suggestions.
>
> A few weeks ago, we reduced the PostgreSQL checkpoint interval to 1
> minute. Could this change contribute to the issue we're seeing?
>
> During our investigation, our infrastructure team identified what appears
> to be an *NVMe storage issue* on the Azure VM. The kernel logs show
> repeated NVMe controller resets followed by multiple read I/O errors across
> several NVMe devices:
>
> kernel: nvme: resetting controller due to persistent internal error
> kernel: Read ... Host Aborted Command
> kernel: I/O error, dev nvme..., op:(READ)
>
> The PostgreSQL error occurred around the same timeframe, which makes us
> suspect the underlying storage issue may have caused the WAL flush failure:
>
> request to flush past end of generated WAL
> xlog flush request ... is not satisfied
>
> Our latest pgBackRest backup is from July 18, as we currently run weekly
> full backups.
>
> Has anyone encountered a similar issue where an NVMe/storage failure
> resulted in PostgreSQL reporting WAL flush errors like this? Is there
> anything else we should be checking from the PostgreSQL side while the
> infrastructure team investigates the storage issue?
>
> On Mon, Jul 27, 2026 at 2:15 PM Gaurav Shukla <gsricky391(at)gmail(dot)com>
> wrote:
>
>> Hi,
>>
>> From the logs, it looks like PostgreSQL is facing an issue while trying
>> to flush WAL during a checkpoint. I would suggest checking a few things:
>>
>> - Check the PostgreSQL logs around the same time to see if there were
>> any other WAL, checkpoint, or I/O-related errors.
>> - Check the server/OS logs for any disk or filesystem issues around 10:58:56
>> MST.
>> - Check if there was any unexpected server or PostgreSQL restart,
>> storage issue, snapshot/rollback, or failover.
>> - Verify the disk and filesystem health and make sure there is enough
>> free space.
>> - Identify which table or index is related to base/16384/1255337, as
>> PostgreSQL was writing to this relation when the error occurred.
>> - Check if WAL archiving and replication are working fine, if
>> configured.
>> - Also, make sure the latest backup is healthy and available for
>> recovery before taking any corrective action.
>>
>> The export issue may just be a symptom of an underlying database or
>> storage problem, so I think these checks would be a good starting point.
>>
>> Thanks & Regards
>> Gaurav Shukla
>> Ph: +91-7022636963
>>
>> On Mon, Jul 27, 2026, 11:27 mahamood hussain <hussain(dot)ieg(at)gmail(dot)com>
>> wrote:
>>
>>> Hi team,
>>>
>>> I need some urgent assistance.
>>>
>>> We're running *PostgreSQL 17.6*, and the application team has reported
>>> an *export issue*. While investigating, I found the following messages
>>> in the PostgreSQL logs:
>>>
>>> 2026-07-26 10:58:52.902 MST [1183899] DETAIL: Parameters: $1 = '1785088732902', $2 = 'ctsserver-11783137027378'
>>> 2026-07-26 10:58:56.136 MST [1529364] LOG: checkpoint starting: time
>>> 2026-07-26 10:58:56.447 MST [1529364] LOG: request to flush past end of generated WAL; request 5D116/1, current position C8F/734E66C0
>>> 2026-07-26 10:58:56.447 MST [1529364] CONTEXT: writing block 4400857 of relation base/16384/1255337
>>> 2026-07-26 10:58:56.448 MST [1529364] ERROR: xlog flush request 5D116/1 is not satisfied --- flushed only to C8F/734E66C0
>>> 2026-07-26 10:58:56.448 MST [1529364] CONTEXT: writing block 4400857 of relation base/16384/1255337
>>>
>>> Has anyone come across this issue before or know what could cause it?
>>> Any guidance on the root cause or recommended troubleshooting steps would
>>> be greatly appreciated.
>>>
>>> This is a *Major Incident (MI)* and requires immediate assistance.
>>>
>>> Thanks in advance.
>>>
>>

In response to

Browse pgsql-admin by date

  From Date Subject
Next Message mahamood hussain 2026-08-06 14:11:24 Urgent !!!! Tables inaccessible postgres v17.6
Previous Message mahamood hussain 2026-07-27 09:01:49 Re: MI - WAL issue