Re: pgsql 8.3.3 "Invalid page header" leads to "FATAL: segment too big"

2010/4/22 Achilleas Mantzios <achill(at)matrix(dot)gatewaynet(dot)com>:
> Hello,
> i have this serious problem in one of our remote vessels. (comm is done by minicom to the remote satelite modem)
> I think that this server was under some sort of constant resets or hardware failures.
> Initially,i had this problem:
> ERROR:  invalid page header in block 672720 of relation "pg_toast_125716009"
>
> This toast table corresponds to a table named "mail_message",
>                                Table "public.mail_message"
>  Column   |       Type        |                         Modifiers
> -----------+-------------------+-----------------------------------------------------------
>  msgno     | mail_msgno_domain | not null default nextval('mail_msgno_sequence'::regclass)
>  msgsource | bytea             |
> Indexes:
>    "mail_message_key" PRIMARY KEY, btree (msgno)
>
> (obviously the TOAST table serves the msgsource varlena) the contents of which is not of vital importance.
> I tried, REINDEXING, with no success, and after that, i tried resetting the said block on disk as per this
> suggestion by Tom here: http://old.nabble.com/invalid-page-header-td11981154.html
>
> i found the oid of the table:
> SELECT tableoid,1 from pg_toast.pg_toast_125716009 order by chunk_id LIMIT 1
>  tableoid  | ?column?
> -----------+----------
>  125716013 |        1
>
> (and just to verify)
> SELECT relname from pg_class where oid=125716013;
>      relname
> --------------------
>  pg_toast_125716009
>
> Then i did: (as i said i do not need the contents of msgsource - yet the contents of msgno are vital)
> dd if=/dev/zero of=/usr/local/var/lib/pgsql/data/base/125714957/125716013 seek=672720 bs=8192 count=1

segment have 1.1GB size maximum. You have to catch in what segment the
faulty block is, and reajust the block value from the error report to
the real one in the good segment.

>
> However, after that, unfortunately i get constant postgresql server restarts with:
> FATAL:  segment too big
> server closed the connection unexpectedly
>        This probably means the server terminated abnormally
>        before or while processing the request.
> The connection to the server was lost. Attempting reset: Succeeded.
>
> Is there anything i can do to savage the situation?
>
> (one of) the hard part here is that i dont have neither physical nor network access to the server
> (only ultra expensive unreliable satellite comms)
>
> Thanks for any hints...
>
> --
> Achilleas Mantzios
>
> --
> Sent via pgsql-admin mailing list (pgsql-admin(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-admin
>

--
Cédric Villemain