Re: Postgres won't start after disk space issue

Cheyne Wallace wrote:
> I have a problem i’m stuck on. Our Postgres server ran out of disk space last night and it fell over.
> We’ve resolved the disk space issue and there is plenty now although Postgres will not start.
>
> When I view the main log it reads:
>
> 2015-02-23 17:49:35 UTC LOG:  redo starts at 75/202674D8
> 2015-02-23 17:49:35 UTC LOG:  incomplete startup packet
> 2015-02-23 17:49:36 UTC FATAL:  the database system is starting up
> 2015-02-23 17:49:36 UTC FATAL:  the database system is starting up
> 2015-02-23 17:49:41 UTC FATAL:  the database system is starting up
> 2015-02-23 17:49:41 UTC LOG:  incomplete startup packet
>
> When I look at ps , I can see that postgres is trying to perform a recovery, although it never
> finishes.
>
> - postgres: startup process   recovering 00000001000000750000002D
>
> I ran an strace on this process to try and see what it’s doing and I see these lines repeating over
> and over again:
>
> read(7, 0x7fff6c82fadf, 1)              = -1 EAGAIN (Resource temporarily unavailable)
> open("pg_clog/01ED", O_RDWR|O_CREAT, 0600) = 27
> lseek(27, 212992, SEEK_SET)             = 212992
> write(27, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192
> fsync(27)                               = 0
> close(27)
>
> When I look under pg_clog I can see that the 01ED file it’s opening here is the most recent one and
> has a different file size to the others in the dir.
> I’m assuming this is the file it was writing too when the disk space ran out and now it’s corrupt or
> something along those lines.
>
> Can anyone please help me figure out how to fix this?  I was told that deleting any of these pg_clog
> files would be bad, how ever I had no idea how to repair it.
>
> p.s I also asked this question here: http://dba.stackexchange.com/questions/93594/postgres-wont-start-
> incomplete-startup-packet

Can you get a stack trace from the startup process?

Yours,
Laurenz Albe