RE: Postgresql error : PANIC: could not locate a valid checkpoint record

Hello there,

Please find my response as below:

* Also, are you able to reproduce the issue?

No we are not able to reproduce at our end. But this is happening occasionally at customer environment that has large DB size.

* I.e. try restoring the database to state before upgrade, try to upgrade again. Does this corruption happen again?

Could you please let us know the process to perform this operations. Can I get some document to do the same.

Thanks,
Prabhakar

From: Mateusz Henicz <mateuszhenicz(at)gmail(dot)com>
Sent: Tuesday, June 21, 2022 5:59 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general(at)lists(dot)postgresql(dot)org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record

[EXTERNAL EMAIL]
Could you also answer on questions asked by Laurenz Albe a few emails back? That could shed some light into your issue. We would know if the upgrade was performed properly and possibly point where the issue can be.
Also, are you able to reproduce the issue? I.e. try restoring the database to state before upgrade, try to upgrade again. Does this corruption happen again?
About pg_resetwal, you are right that it should be done as a last resort. On the other hand, if your production database does not start after upgrade and you do not have any way to rollback your changes it might be it.

Cheers,
Mateusz

wt., 21 cze 2022 o 14:02 Mahendrakar, Prabhakar - Dell Team <Prabhakar(dot)Mahendraka(at)dellteam(dot)com<mailto:Prabhakar(dot)Mahendraka(at)dellteam(dot)com>> napisał(a):
Could you please provide some insights in the below query. We are in need of this as it this issue is seen at some of our customer's site.

Thanks,
Prabhakar

Internal Use - Confidential

Internal Use - Confidential
From: Mahendrakar, Prabhakar - Dell Team
Sent: Monday, June 20, 2022 4:22 PM
To: Mateusz Henicz
Cc: pgsql-general(at)lists(dot)postgresql(dot)org<mailto:pgsql-general(at)lists(dot)postgresql(dot)org>
Subject: RE: Postgresql error : PANIC: could not locate a valid checkpoint record

Thanks for the response.
Yes, we have taken care of proper shut down of Postgres before initiating the Upgrade.
pg_resetwal - I have read that using pg_resetwal may cause the Database more inconsistent and should be used only as a last resort.

Also this problem ( checkpoint related issue -could not locate a valid checkpoint record ) is not happening frequently. This issue is seen with the large size of Data base.
Please let me know if you require any more information.

Thanks,
Prabhakar

From: Mateusz Henicz <mateuszhenicz(at)gmail(dot)com<mailto:mateuszhenicz(at)gmail(dot)com>>
Sent: Friday, June 17, 2022 3:39 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general(at)lists(dot)postgresql(dot)org<mailto:pgsql-general(at)lists(dot)postgresql(dot)org>
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record

[EXTERNAL EMAIL]
Assuming you have shut down your postgres properly before upgrading, it should be safe for you to run pg_resetwal.
https://www.postgresql.org/docs/current/app-pgresetwal.html [postgresql.org]<https://urldefense.com/v3/__https:/www.postgresql.org/docs/current/app-pgresetwal.html__;!!LpKI!nLsPoYLxz9_0wbkgjbQGXorp1s6PcSTlir-of_nvRmQZazd2S0Rnqe7iZ8l6oW7aarWf8v4_p1LdgHThZk0YSPfaD4cz4g-K$>
It should help in this case.

pt., 17 cze 2022 o 12:03 Mahendrakar, Prabhakar - Dell Team <Prabhakar(dot)Mahendraka(at)dellteam(dot)com<mailto:Prabhakar(dot)Mahendraka(at)dellteam(dot)com>> napisał(a):
Yes, We are using the pg_upgrade utility of Postgres.

From: Mateusz Henicz <mateuszhenicz(at)gmail(dot)com<mailto:mateuszhenicz(at)gmail(dot)com>>
Sent: Friday, June 17, 2022 3:31 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general(at)lists(dot)postgresql(dot)org<mailto:pgsql-general(at)lists(dot)postgresql(dot)org>
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record

[EXTERNAL EMAIL]
Hi,
Have you done pg_upgrade post Postgres 13 installation?

https://www.postgresql.org/docs/13/pgupgrade.html [postgresql.org]<https://urldefense.com/v3/__https:/www.postgresql.org/docs/13/pgupgrade.html__;!!LpKI!idvJ795EwLEkEftyMMQc6j4AvHBN-AWOsHhukZYHV0VBPR8PRtDCRs_JDRGs5Nz34Vs4VK05szBLxdJ13Wzjztu6q67YUhUY$>

Cheers,
Mateusz

pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <Prabhakar(dot)Mahendraka(at)dellteam(dot)com<mailto:Prabhakar(dot)Mahendraka(at)dellteam(dot)com>> napisał(a):
Hello,

Good Morning !

We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.
========================================================================================================
LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG: listening on IPv4 address "127.0.0.1", port 9003
LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG: database system was shut down at 2022-06-09 10:19:24 CEST
LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 8773) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down

This issue is seen in both Windows and Linux OS platforms.

To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.

Could you please suggest the probable cause of the issue. Let us know if you require any more information.

Thanks,
Prabhakar

Internal Use - Confidential

Internal Use - Confidential

Internal Use - Confidential