Immediate shutdown and system(3)

We're using SIGQUIT to signal immediate shutdown request. Upon receiving
SIGQUIT, postmaster in turn kills all the child processes with SIGQUIT
and exits.

This is a problem when child processes use system(3) to call other
programs. We use system(3) in two places: to execute archive_command and
restore_command. Fujii Masao identified this with pg_standby back in
November:

http://archives.postgresql.org/message-id/3f0b79eb0811280156s78a3730en73aca49b6e95d3cb@mail.gmail.com
and recently discussed here
http://archives.postgresql.org/message-id/3f0b79eb0902260919l2675aaafq10e5b2d49ebfa3a1@mail.gmail.com

I'm starting a new thread to bring this to attention of those who
haven't been following the hot standby stuff. pg_standby has a
particular problem because it traps SIGQUIT to mean "end recovery,
promote standby to master", which it shouldn't do IMHO. But ignoring
that for a moment, the problem is generic.

SIGQUIT by default dumps core. That's not what we want to happen on
immediate shutdown. All PostgreSQL processes trap SIGQUIT to exit
immediately instead, but external commands will dump core. system(3)
ignores SIGQUIT, so we can't trap it in the parent process; it is always
relayed to the child.

There's a few options on how to fix that:

1. Implement a custom version of system(3) using fork+exec that let's us
trap SIGQUIT and send e.g SIGTERM or SIGINT to the child instead. It
might be a bit tricky to get this right in a portable way; Windows would
certainly need a completely separate implementation.

2. Use a signal other than SIGQUIT for immediate shutdown of child
processes. We can't change the signal sent to postmaster for
backwards-compatibility reasons, but the signal sent by postmaster to
child processes we could change. We've already used all signals in
normal backends, but perhaps we could rearrange them.

3. Use SIGINT instead of SIGQUIT for immediate shutdown of the two child
processes that use system(3): the archiver process and the startup
process. Neither of them use SIGINT currently. SIGINT is ignored by
system(3), like SIGQUIT, but the default action is to terminate the
process rather than core dump. Unfortunately pg_standby traps SIGINT too
to mean "promote to master", but we could change it to use SIGUSR1
instead for that purpose. If someone has a script that uses "killall
-INT pg_standby" to promote a standby server to master, it would need to
be changed. Looking at the manual page of pg_standby, however, it seems
that the kill-method of triggering a promotion isn't documented, so with
a notice in release notes we could do that.

I'm leaning towards option 3, but I wonder if anyone sees a better solution.

This is all for CVS HEAD. In back-branches, I think we should just
remove the signal handler for SIGQUIT from pg_standby and leave it at
that. If you perform an immediate shutdown, you can get a core dump from
archive_command or restore_command, but that's a minor inconvenience.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com