corruption issue after server crash - ERROR: unexpected chunk number 0

From: Mike Broers <mbroers(at)gmail(dot)com>
To: pgsql-general(at)postgresql(dot)org
Subject: corruption issue after server crash - ERROR: unexpected chunk number 0
Date: 2013-11-21 20:10:56
Message-ID: CAB9893jbkVdmLUZBnk25Kan8WevGbhz6gGq-TT1Zdb7pEe=CRw@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Hello we are running postgres 9.2.5 on RHEL6, our production server crashed
hard and when it came back up our logs were flooded with:

STATEMENT: SELECT "session_session"."session_key",
"session_session"."session_data", "session_session"."expire_date",
"session_session"."nonce" FROM "session_session" WHERE
("session_session"."session_key" = 'gk9aap5d7btp6tzquh0kf73gpfmik5w3' AND
"session_session"."expire_date" > '2013-11-21 13:27:33.107913' )

ERROR: unexpected chunk number 0 (expected 1) for toast value 117927127 in
pg_toast_19122

We restarted the application and whatever session was constantly hitting
that row stopped, but Im concerned about remediation. When I attempt to
read from that row the error occurs.

select * from session_session where session_key =
'gk9aap5d7btp6tzquh0kf73gpfmik5w3';
ERROR: unexpected chunk number 0 (expected 1) for toast value 117927127 in
pg_toast_19122

When I attempt to delete this row I get this error:
delete from session_session where session_key =
'gk9aap5d7btp6tzquh0kf73gpfmik5w3';
ERROR: tuple concurrently updated

We happen to have a maintenance window tonight so I will have some time
when the app is down to run some database fixes. I saw other threads
suggesting a reindex of the toast table, but this is a 14GB table and I'm
not sure how long that will take or if it will even be successful. We also
have a full db vacuum/analyze scheduled nightly for 2am so I am expecting
to learn if there are other impacted tables, but its troubling if I dont
know what the remediation. This particular table could be truncated if
necessary if that is an option but Im not sure about other tables.

Any suggestions for how to handle the tuple concurrently updated error? Or
if a reindex is likely to help with the unexpected chunk error?

Thanks
Mike

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Joey Quinn 2013-11-21 20:40:34 Primary Key
Previous Message Karsten Hilbert 2013-11-21 14:39:18 Re: pg_upgrade ?deficiency