Re: production server down

Tom Lane wrote:
> Are you using one of the scripts that
> does an auto initdb if it doesn't see a valid PGDATA? 11 seconds might
> be about right for that.
>
> One problem with this theory is how come you didn't get screwed during
> *that* boot cycle. It seems to require assuming that the NFS mount came
> online just after the initdb finished (else initdb would have
> overwritten the on-NFS pg_control) but before the regular postmaster
> started (else this same scenario would have played out then). That's
> not a very wide window.

[followup]
We've now had a chance to bring Postgres down and check under the mount
point. There *is* indeed a newly initdb'd cluster under there. FWIW the
control file is corrupt:

# pg_controldata /home/jconway/pgsql/fds/replica/pgdata
WARNING: Calculated CRC checksum does not match value stored in file.
Either the file is corrupt, or it has a different layout than this program
is expecting. The results below are untrustworthy.

pg_control version number: 72
Catalog version number: 200310211
Database cluster state: in production
pg_control last modified: Sat Feb 6 22:28:16 2106
Current log file ID: 0
Next log file segment: 10161036
Latest checkpoint location: 0/9AA1B4
Prior checkpoint location: 0/9B0B8C
Latest checkpoint's REDO location: 0/0
Latest checkpoint's UNDO location: C/218
Latest checkpoint's StartUpID: 17142
Latest checkpoint's NextXID: 1099443932
Latest checkpoint's NextOID: 8192
Time of latest checkpoint: Wed Apr 8 07:05:36 6325
Database block size: 1
Blocks per segment of large relation: 128
Maximum length of identifiers: 67
Maximum number of function arguments: 0
Date/time type storage: floating-point numbers
Maximum length of locale name: 0
LC_COLLATE:
LC_CTYPE:

I have a tarred copy of the under-the-mount PGDATA if anyone is
interested in examining it.

BTW, there was another Postgres cluster on this same server which we had
not used since the November 2 reboot -- it was corrupt in pretty much
the same way and also had an initdb'd cluster under its mount.

So it looks like using an auto initdb startup script is a very bad idea
when using an NFS mounted PGDATA. We left the under-mount structure in
place and did "chown root:root" and "chmod 000" on it. And, as mentioned
in an earlier post, we now rely on the dba to start postgres manually
after a server restart.

Joe