Re: Urgent !!!! Tables inaccessible postgres v17.6

hi Matos,

Quick question regarding disk striping.

The main reason we considered disk striping was to keep the storage cost
reasonable while achieving the required IOPS and throughput. We are using
Azure Premium SSD v2, which provides 99.999999999% durability, and my
understanding is that Azure designed this storage to support configurations
such as RAID 0 when higher IOPS/throughput are required.

Given that our databases are in the TB range—the current database is around
13 TB—are there any known limitations or risks with using RAID 0 even with
the underlying durability provided by Premium SSD v2?

I understand that traditionally we would avoid RAID 0 from a
hardware/database reliability perspective because a failure of a single
disk can impact the entire volume. However, with Azure managed disks and
their durability guarantees, I'm trying to understand whether the same
concern still applies and what the recommended architecture is.

If RAID 0 is not the preferred approach, what is the recommended PostgreSQL
architecture for achieving high IOPS and throughput for large databases?

For context, these databases were recently migrated from DB2 to PostgreSQL.
In DB2, we have the concept of storage groups and storage paths, which
allows us to place tablespaces across multiple disks/storage paths for
performance and I/O distribution.

Since PostgreSQL doesn't have the same storage-group concept, I was
considering implementing a similar architecture at the storage layer using
RAID 0.

I'd appreciate your thoughts on the recommended approach for large
PostgreSQL databases on Azure, particularly around performance, cost, and
reliability.

[postgres(at)dtprd04-pg01: ~]$ dmesg -T | grep -iE 'error|reset|i/o'
[Fri Jul 31 23:44:38 2026] APIC: Switch to symmetric I/O mode setup
[Fri Jul 31 23:44:39 2026] 00:00: ttyS0 at I/O 0x3f8 (irq = 4, base_baud =
115200) is a 16550A
[Fri Jul 31 23:44:39 2026] 00:01: ttyS1 at I/O 0x2f8 (irq = 3, base_baud =
115200) is a 16550A
[Fri Jul 31 23:44:39 2026] serial8250: ttyS2 at I/O 0x3e8 (irq = 4,
base_baud = 115200) is a 16550A
[postgres(at)dtprd04-pg01: ~]$ sudo journalctl -k --since "10 days ago" | grep
-iE 'ata|nvme|scsi'
Jul 31 23:44:39 localhost kernel: The list of certified hardware and cloud
instances for Red Hat Enterprise Linux 9 can be viewed at the Red Hat
Ecosystem Catalog, https://catalog.redhat.com.
Jul 31 23:44:39 localhost kernel: Command line:
BOOT_IMAGE=(hd2,gpt3)/vmlinuz-5.14.0-687.26.1.el9_8.x86_64
root=UUID=e8698ddb-90ea-4097-a275-643af6bbabb8 ro loglevel=3 console=tty1
console=ttyS0 earlyprintk=ttyS0 rootdelay=300 no_timer_check
nvme_core.io_timeout=240 biosdevname=0 net.ifnames=0
crashkernel=1G-2G:192M,2G-64G:256M,64G-:512M
Jul 31 23:44:39 localhost kernel: BIOS-e820: [mem
0x000000003ffc9000-0x000000003fffafff] ACPI data
Jul 31 23:44:39 localhost kernel: NODE_DATA(0) allocated [mem
0x60fffd5000-0x60ffffffff]
Jul 31 23:44:39 localhost kernel: Kernel command line:
BOOT_IMAGE=(hd2,gpt3)/vmlinuz-5.14.0-687.26.1.el9_8.x86_64
root=UUID=e8698ddb-90ea-4097-a275-643af6bbabb8 ro loglevel=3 console=tty1
console=ttyS0 earlyprintk=ttyS0 rootdelay=300 no_timer_check
nvme_core.io_timeout=240 biosdevname=0 net.ifnames=0
crashkernel=1G-2G:192M,2G-64G:256M,64G-:512M
Jul 31 23:44:39 localhost kernel: Memory: 395392972K/402647804K available
(16384K kernel code, 5797K rwdata, 13972K rodata, 4208K init, 7184K bss,
7228844K reserved, 0K cma-reserved)
Jul 31 23:44:39 localhost kernel: SCSI subsystem initialized
Jul 31 23:44:39 localhost kernel: Block layer SCSI generic (bsg) driver
version 0.4 loaded (major 246)
Jul 31 23:44:39 localhost kernel: Write protecting the kernel read-only
data: 30720k
Jul 31 23:44:39 localhost kernel: Freeing unused kernel image (rodata/data
gap) memory: 364K
Jul 31 23:44:40 localhost kernel: scsi host0: storvsc_host_t
Jul 31 23:44:40 localhost kernel: nvme nvme0: pci function c05b:00:00.0
Jul 31 23:44:40 localhost kernel: nvme c05b:00:00.0: enabling device (0000
-> 0002)
Jul 31 23:44:40 localhost kernel: nvme nvme1: pci function 8d07:00:00.0
Jul 31 23:44:40 localhost kernel: nvme nvme2: pci function e4d7:00:00.0
Jul 31 23:44:40 localhost kernel: nvme nvme3: pci function 096c:00:00.0
Jul 31 23:44:40 localhost kernel: nvme nvme4: pci function f0bc:00:00.0
Jul 31 23:44:40 localhost kernel: nvme nvme5: pci function 033d:00:00.0
Jul 31 23:44:40 localhost kernel: nvme 8d07:00:00.0: enabling device (0000
-> 0002)
Jul 31 23:44:40 localhost kernel: nvme nvme6: pci function b7bb:00:00.0
Jul 31 23:44:40 localhost kernel: nvme e4d7:00:00.0: enabling device (0000
-> 0002)
Jul 31 23:44:40 localhost kernel: nvme f0bc:00:00.0: enabling device (0000
-> 0002)
Jul 31 23:44:40 localhost kernel: nvme 096c:00:00.0: enabling device (0000
-> 0002)
Jul 31 23:44:40 localhost kernel: nvme 033d:00:00.0: enabling device (0000
-> 0002)
Jul 31 23:44:40 localhost kernel: nvme b7bb:00:00.0: enabling device (0000
-> 0002)
Jul 31 23:44:40 localhost kernel: nvme nvme0: 48/0/0 default/read/poll
queues
Jul 31 23:44:40 localhost kernel: nvme0n1: p1 p2 p3 p4
Jul 31 23:44:40 localhost kernel: nvme nvme6: 6/0/0 default/read/poll queues
Jul 31 23:44:41 localhost kernel: nvme nvme1: 6/0/0 default/read/poll queues
Jul 31 23:44:41 localhost kernel: nvme nvme4: 6/0/0 default/read/poll queues
Jul 31 23:44:41 localhost kernel: nvme nvme2: 6/0/0 default/read/poll queues
Jul 31 23:44:41 localhost kernel: nvme nvme3: 6/0/0 default/read/poll queues
Jul 31 23:44:41 localhost kernel: nvme nvme5: 6/0/0 default/read/poll queues
Jul 31 23:44:42 localhost kernel: XFS (nvme0n1p4): Mounting V5 Filesystem
e8698ddb-90ea-4097-a275-643af6bbabb8
Jul 31 23:44:42 localhost kernel: XFS (nvme0n1p4): Ending clean mount
Jul 31 23:44:44 dtprd04-pg01 kernel: XFS (nvme0n1p3): Mounting V5
Filesystem 050b8d52-f03b-4fbc-87b3-fee63a357e2c
Jul 31 23:44:44 dtprd04-pg01 kernel: XFS (nvme0n1p3): Ending clean mount
Jul 31 23:44:44 dtprd04-pg01 kernel: XFS (nvme0n13): Mounting V5 Filesystem
ff919446-a2bb-459d-a7d2-6a894a5656e1
Jul 31 23:44:45 dtprd04-pg01 kernel: XFS (nvme0n13): Ending clean mount
Jul 31 23:44:46 dtprd04-pg01 kernel: hv_netvsc
f8615163-0000-1000-2000-7c1e52d8883d eth0: Data path switched to VF: eth1
Jul 31 23:44:47 dtprd04-pg01 kernel: hv_netvsc
f8615163-0000-1000-2000-7c1e52d8883d eth0: Data path switched from VF: eth1
Jul 31 23:44:47 dtprd04-pg01 kernel: hv_netvsc
f8615163-0000-1000-2000-7c1e52d8883d eth0: Data path switched to VF: eth1
Jul 31 23:44:51 dtprd04-pg01 kernel: block nvme0n1: No UUID available
providing old NGUID
Jul 31 23:44:51 dtprd04-pg01 kernel: block nvme0n1: the capability
attribute has been deprecated.
Aug 08 06:21:45 dtprd04-pg01 kernel: nvme nvme0: rescanning namespaces.

SHOW full_page_writes;
full_page_writes
------------------
on
(1 row)

On Mon, Aug 10, 2026 at 6:09 PM Licio Matos <licio(dot)matos(at)gmail(dot)com> wrote:

> Mahamood,
>
> Understand, so there some issues regarding ssd v2.
>
> When you stripe 4 Premium SSD v2 disks together with LVM or mdadm, the OS
> splits each logical I/O into chunks and distributes them across the 4
> physical disks based on offset. A single PostgreSQL page write (8KB) can
> end up physically split across 2, 3, or all 4 disks, depending on where it
> lands relative to the stripe’s chunk size.
>
> The danger: if even one of those four disks is momentarily throttled —
> because it hit its own individually-provisioned IOPS/throughput ceiling —
> that disk’s portion of the write lags behind while the other three complete
> on time. What was meant to be one atomic 8KB write becomes several
> independent physical operations with different completion timing. If a
> crash or reset happens in that window, you get a torn write: part of the
> page has the new data, part still has old or garbage data. That’s exactly
> the signature of a checksum mismatch like the one you hit.
>
> For digging in the problem, can you provide this informations:
>
> Run this with Az CLI:
>
> az disk show -g <rg> -n <disk-name> --query
> "diskIOPSReadWrite,diskMBpsReadWrite,networkAccessPolicy"
>
> Check full page writes are ON.
>
> SHOW full_page_writes;
>
> If you have access to the VM. You could look at the journal with dmesg
> checking for IO errors:
>
> dmesg -T | grep -iE 'error|reset|i/o'
> journalctl -k --since "3 days ago" | grep -iE 'ata|nvme|scsi'
>
>
>
> Licio Matos
>
> Em seg., 10 de ago. de 2026 às 01:23, mahamood hussain <
> hussain(dot)ieg(at)gmail(dot)com> escreveu:
>
>> Hi,
>>
>> We were using Premium SSD v2, which Microsoft states provides
>> 99.999999999% durability. However, given the corruption issue we
>> encountered, I’m not fully confident in the storage layer.
>>
>> The corrupted page was a heap/table page, not an index page.
>>
>> For recovery, I restored the database backup on a temporary server and
>> performed a roll-forward until it caught up with production. I then
>> exported the recovered table data, imported it into production, and
>> replaced the corrupted table with the recovered copy.
>>
>> I have not dropped the original corrupted table yet. I have kept it
>> renamed for now so that we can investigate the corruption further and try
>> some of the other recommended recovery methods before removing it.
>> On Sat, Aug 8, 2026 at 3:38 AM Licio Matos <licio(dot)matos(at)gmail(dot)com> wrote:
>>
>>>
>>> Have you try to check these blocks are table related or index related?
>>>
>>> You are counting, could be a index corrupted.
>>>
>>> Try this:
>>>
>>> SELECT relname, relkind
>>> FROM pg_class
>>> WHERE pg_relation_filepath(oid) LIKE '%447758%';
>>>
>>> Licio Matos
>>>
>>> Em sex., 7 de ago. de 2026 às 18:56, Laurenz Albe <
>>> laurenz(dot)albe(at)cybertec(dot)at> escreveu:
>>>
>>>> On Thu, 2026-08-06 at 19:57 +0530, mahamood hussain wrote:
>>>> > The database is hosted on Azure Premium SSD v2, using four striped
>>>> disks.
>>>> >
>>>> > >
>>>> > > > prod=# SELECT count(*) FROM schema.tablename;
>>>> > > >
>>>> > > > WARNING: page verification failed, calculated checksum 50897 but
>>>> expected 50048
>>>> > > > ERROR: invalid page in block 696770 of relation
>>>> base/16388/447758
>>>>
>>>> Looks like Microsoft's storage is not reliable.
>>>>
>>>> Yours,
>>>> Laurenz Albe
>>>>
>>>>
>>>>