Re: Postgres server goes in recovery mode repeteadly

On Tue, Sep 29, 2009 at 09:52:06PM +0530, kunal sharma wrote:
> Hi ,
> We are using Postgres 8.4 and its been found going into recovery
> mode couple of times. The server process seems to fork another child process
> which is another postgres server running under same data directory and after
> some time it goes away while the old server is still running. There were few
> load issues on the server but the load didnt went above "32".
>
> We are running opensuse 10.2 x86_64 with 32Gb of physical memory.
> Checking the logs I found that theres a segmentation fault ,
>
>
> Sep 26 05:39:54 pace kernel: postgres[28694]: segfault at 0000000000000030
> rip 000000000066ba8c rsp 00007fffd364da30 error 4
>
> gdb dump shows this
>
> Reading symbols from /lib64/libdl.so.2...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libm.so.6...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libnss_files.so.2...done.
> Loaded symbols for /lib64/libnss_files.so.2
> 0x00002ad6d7b8c2b3 in __select_nocancel () from /lib64/libc.so.6
> (gdb)
>
> Any suggestions what is causing this segmentation fault?

I work with Kunal and have been looking into this. It appears to be the same
as the bug described in:

http://archives.postgresql.org/pgsql-bugs/2009-09/msg00355.php

as I have localized it to a NULL pointer deference in
RelationCacheInitializePhase2() as well. Tom speculates in:

http://archives.postgresql.org/pgsql-bugs/2009-09/msg00372.php

that large numbers of table drops might trigger this. The system in question
creates and drops temp tables at a high rate which tends to confirm this.

-dg

--
David Gould daveg(at)sonic(dot)net 510 536 1443 510 282 0869
If simplicity worked, the world would be overrun with insects.