Re: "PANIC: could not open critical system index 2662" - twice

On Thu, 2023-04-06 at 16:41 +0000, Evgeny Morozov wrote:
>  Our PostgreSQL 15.2 instance running on Ubuntu 18.04 has crashed with this error:
>
> 2023-04-05 09:24:03.448 UTC [15227] ERROR:  index "pg_class_oid_index" contains unexpected zero page at block 0
> [...]
>
> We had the same thing happened about a month ago on a different database on the same cluster.
> For a while PG actually ran OK as long as you didn't access that specific DB, but when trying
> to back up that DB with pg_dump it would crash every time. At that time one of the disks
> hosting the ZFS dataset with the PG data directory on it was reporting errors, so we thought
> it was likely due to that.
>
> Unfortunately, before we could replace the disks, PG crashed completely and would not start
> again at all, so I had to rebuild the cluster from scratch and restore from pg_dump backups
> (still onto the old, bad disks). Once the disks were replaced (all of them) I just copied
> the data to them using zfs send | zfs receive and didn't bother restoring pg_dump backups
> again - which was perhaps foolish in hindsight.
>
> Well, yesterday it happened again. The server still restarted OK, so I took fresh pg_dump
> backups of the databases we care about (which ran fine), rebuilt the cluster and restored
> the pg_dump backups again - now onto the new disks, which are not reporting any problems.
>
> So while everything is up and running now this error has me rather concerned. Could the
> error we're seeing now have been caused by some corruption in the PG data that's been there
> for a month (so it could still be attributed to the bad disk), which should now be fixed by
> having restored from backups onto good disks?

Yes, that is entirely possible.

> Could this be a PG bug?

It could be, but data corruption caused by bad hardware is much more likely.

> What can I do to figure out why this is happening and prevent it from happening again?

No idea about the former, but bad hardware is a good enough explanation.

As to keeping it from happening: use good hardware.

Yours,
Laurenz Albe