| From: | sayli khedekar <saylikhedekar1(at)gmail(dot)com> |
|---|---|
| To: | mahamood hussain <hussain(dot)ieg(at)gmail(dot)com> |
| Cc: | Pgsql-admin <pgsql-admin(at)lists(dot)postgresql(dot)org> |
| Subject: | Re: MI - WAL issue |
| Date: | 2026-07-27 11:38:10 |
| Message-ID: | CAEUYW+T3EPLAt1Y8qC=tmuipRGzBJEof6sQz9-p_Vv6cE9NTYQ@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-admin |
Hi ,
Can't confirm the checkpoint_timeout change directly caused the WAL flush
error, but the 1-min checkpoint interval may add extra write I/O -- two
things worth checking to reduce that load:
1) HOT update ratio + fillfactor — for busiest tables, check
n_tup_hot_upd/n_tup_upd in pg_stat_user_tables. If low (and not due to
indexed columns changing), lower fillfactor to 70–90 to allow more HOT
updates.
2) Unused/duplicate indexes — check pg_stat_user_indexes for near-zero
idx_scan and check for duplicates on the same columns.
pg_wal size may also have increased due to more full page image, but I
don't see its relevance here.
Regards,
Sayli
On Mon, Jul 27, 2026 at 2:32 PM mahamood hussain <hussain(dot)ieg(at)gmail(dot)com>
wrote:
> Thanks for the suggestions.
>
> A few weeks ago, we reduced the PostgreSQL checkpoint interval to 1
> minute. Could this change contribute to the issue we're seeing?
>
> During our investigation, our infrastructure team identified what appears
> to be an *NVMe storage issue* on the Azure VM. The kernel logs show
> repeated NVMe controller resets followed by multiple read I/O errors across
> several NVMe devices:
>
> kernel: nvme: resetting controller due to persistent internal error
> kernel: Read ... Host Aborted Command
> kernel: I/O error, dev nvme..., op:(READ)
>
> The PostgreSQL error occurred around the same timeframe, which makes us
> suspect the underlying storage issue may have caused the WAL flush failure:
>
> request to flush past end of generated WAL
> xlog flush request ... is not satisfied
>
> Our latest pgBackRest backup is from July 18, as we currently run weekly
> full backups.
>
> Has anyone encountered a similar issue where an NVMe/storage failure
> resulted in PostgreSQL reporting WAL flush errors like this? Is there
> anything else we should be checking from the PostgreSQL side while the
> infrastructure team investigates the storage issue?
>
> On Mon, Jul 27, 2026 at 2:15 PM Gaurav Shukla <gsricky391(at)gmail(dot)com>
> wrote:
>
>> Hi,
>>
>> From the logs, it looks like PostgreSQL is facing an issue while trying
>> to flush WAL during a checkpoint. I would suggest checking a few things:
>>
>> - Check the PostgreSQL logs around the same time to see if there were
>> any other WAL, checkpoint, or I/O-related errors.
>> - Check the server/OS logs for any disk or filesystem issues around 10:58:56
>> MST.
>> - Check if there was any unexpected server or PostgreSQL restart,
>> storage issue, snapshot/rollback, or failover.
>> - Verify the disk and filesystem health and make sure there is enough
>> free space.
>> - Identify which table or index is related to base/16384/1255337, as
>> PostgreSQL was writing to this relation when the error occurred.
>> - Check if WAL archiving and replication are working fine, if
>> configured.
>> - Also, make sure the latest backup is healthy and available for
>> recovery before taking any corrective action.
>>
>> The export issue may just be a symptom of an underlying database or
>> storage problem, so I think these checks would be a good starting point.
>>
>> Thanks & Regards
>> Gaurav Shukla
>> Ph: +91-7022636963
>>
>> On Mon, Jul 27, 2026, 11:27 mahamood hussain <hussain(dot)ieg(at)gmail(dot)com>
>> wrote:
>>
>>> Hi team,
>>>
>>> I need some urgent assistance.
>>>
>>> We're running *PostgreSQL 17.6*, and the application team has reported
>>> an *export issue*. While investigating, I found the following messages
>>> in the PostgreSQL logs:
>>>
>>> 2026-07-26 10:58:52.902 MST [1183899] DETAIL: Parameters: $1 = '1785088732902', $2 = 'ctsserver-11783137027378'
>>> 2026-07-26 10:58:56.136 MST [1529364] LOG: checkpoint starting: time
>>> 2026-07-26 10:58:56.447 MST [1529364] LOG: request to flush past end of generated WAL; request 5D116/1, current position C8F/734E66C0
>>> 2026-07-26 10:58:56.447 MST [1529364] CONTEXT: writing block 4400857 of relation base/16384/1255337
>>> 2026-07-26 10:58:56.448 MST [1529364] ERROR: xlog flush request 5D116/1 is not satisfied --- flushed only to C8F/734E66C0
>>> 2026-07-26 10:58:56.448 MST [1529364] CONTEXT: writing block 4400857 of relation base/16384/1255337
>>>
>>> Has anyone come across this issue before or know what could cause it?
>>> Any guidance on the root cause or recommended troubleshooting steps would
>>> be greatly appreciated.
>>>
>>> This is a *Major Incident (MI)* and requires immediate assistance.
>>>
>>> Thanks in advance.
>>>
>>
| From | Date | Subject | |
|---|---|---|---|
| Next Message | mahamood hussain | 2026-08-06 14:11:24 | Urgent !!!! Tables inaccessible postgres v17.6 |
| Previous Message | mahamood hussain | 2026-07-27 09:01:49 | Re: MI - WAL issue |