| From: | mahamood hussain <hussain(dot)ieg(at)gmail(dot)com> |
|---|---|
| To: | Gaurav Shukla <gsricky391(at)gmail(dot)com> |
| Cc: | Pgsql-admin <pgsql-admin(at)lists(dot)postgresql(dot)org> |
| Subject: | Re: MI - WAL issue |
| Date: | 2026-07-27 09:01:49 |
| Message-ID: | CAGc_7HmK=ONV6g0q_ExtLTUq07UEP+j=dBntuKFL+A1UGKa_jQ@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-admin |
Thanks for the suggestions.
A few weeks ago, we reduced the PostgreSQL checkpoint interval to 1 minute.
Could this change contribute to the issue we're seeing?
During our investigation, our infrastructure team identified what appears
to be an *NVMe storage issue* on the Azure VM. The kernel logs show
repeated NVMe controller resets followed by multiple read I/O errors across
several NVMe devices:
kernel: nvme: resetting controller due to persistent internal error
kernel: Read ... Host Aborted Command
kernel: I/O error, dev nvme..., op:(READ)
The PostgreSQL error occurred around the same timeframe, which makes us
suspect the underlying storage issue may have caused the WAL flush failure:
request to flush past end of generated WAL
xlog flush request ... is not satisfied
Our latest pgBackRest backup is from July 18, as we currently run weekly
full backups.
Has anyone encountered a similar issue where an NVMe/storage failure
resulted in PostgreSQL reporting WAL flush errors like this? Is there
anything else we should be checking from the PostgreSQL side while the
infrastructure team investigates the storage issue?
On Mon, Jul 27, 2026 at 2:15 PM Gaurav Shukla <gsricky391(at)gmail(dot)com> wrote:
> Hi,
>
> From the logs, it looks like PostgreSQL is facing an issue while trying to
> flush WAL during a checkpoint. I would suggest checking a few things:
>
> - Check the PostgreSQL logs around the same time to see if there were
> any other WAL, checkpoint, or I/O-related errors.
> - Check the server/OS logs for any disk or filesystem issues around 10:58:56
> MST.
> - Check if there was any unexpected server or PostgreSQL restart,
> storage issue, snapshot/rollback, or failover.
> - Verify the disk and filesystem health and make sure there is enough
> free space.
> - Identify which table or index is related to base/16384/1255337, as
> PostgreSQL was writing to this relation when the error occurred.
> - Check if WAL archiving and replication are working fine, if
> configured.
> - Also, make sure the latest backup is healthy and available for
> recovery before taking any corrective action.
>
> The export issue may just be a symptom of an underlying database or
> storage problem, so I think these checks would be a good starting point.
>
> Thanks & Regards
> Gaurav Shukla
> Ph: +91-7022636963
>
> On Mon, Jul 27, 2026, 11:27 mahamood hussain <hussain(dot)ieg(at)gmail(dot)com>
> wrote:
>
>> Hi team,
>>
>> I need some urgent assistance.
>>
>> We're running *PostgreSQL 17.6*, and the application team has reported
>> an *export issue*. While investigating, I found the following messages
>> in the PostgreSQL logs:
>>
>> 2026-07-26 10:58:52.902 MST [1183899] DETAIL: Parameters: $1 = '1785088732902', $2 = 'ctsserver-11783137027378'
>> 2026-07-26 10:58:56.136 MST [1529364] LOG: checkpoint starting: time
>> 2026-07-26 10:58:56.447 MST [1529364] LOG: request to flush past end of generated WAL; request 5D116/1, current position C8F/734E66C0
>> 2026-07-26 10:58:56.447 MST [1529364] CONTEXT: writing block 4400857 of relation base/16384/1255337
>> 2026-07-26 10:58:56.448 MST [1529364] ERROR: xlog flush request 5D116/1 is not satisfied --- flushed only to C8F/734E66C0
>> 2026-07-26 10:58:56.448 MST [1529364] CONTEXT: writing block 4400857 of relation base/16384/1255337
>>
>> Has anyone come across this issue before or know what could cause it? Any
>> guidance on the root cause or recommended troubleshooting steps would be
>> greatly appreciated.
>>
>> This is a *Major Incident (MI)* and requires immediate assistance.
>>
>> Thanks in advance.
>>
>
| From | Date | Subject | |
|---|---|---|---|
| Next Message | sayli khedekar | 2026-07-27 11:38:10 | Re: MI - WAL issue |
| Previous Message | Gaurav Shukla | 2026-07-27 08:45:23 | Re: MI - WAL issue |