Re: Better client reporting for "immediate stop" shutdowns

On Sat, Dec 26, 2020 at 4:33 AM Andres Freund <andres(at)anarazel(dot)de> wrote:
> On 2020-12-21 16:43:33 -0500, Tom Lane wrote:
> > * I chose to report the same message for immediate shutdown as we
> > already use for SIGTERM (fast shutdown or pg_terminate_backend()).
> > Should it be different, and if so what?
>
> To do better I think we'd have to distinguish the different cases? An
> error message like
> "terminating connection due to {fast shutdown,immediate shutdown,connection termination} administrator command"
> or such could be helpful, but I don't think your patch adds *quite*
> enough state?

Currently, for fast shutdown, the "FATAL: terminating connection due
to administrator command" message is shown in server logs per backend.
The idea used for immediate shutdown can be extended to fast shutdown
as well, that is postmaster can set the signal state just before
signalling the backends with SIGTERM and later in ProcessInterrupts()
the status can be checked and report something like "FATAL:
terminating connection due to fast shutdown command".

And for smart shutdown, since the postmaster waits until the normal
backends to go away on their own and no FATAL messages get logged, so
we don't need to set the signal state.

> I'd like to not log all these repeated messages into the server
> log. It's quite annoying to have to digg through thousands of lines of
> repeated "terminating connection..." lines that add absolutely no
> additional information, just because I am shutting down the
> server. Similarly, trying to find the reason for a PANIC is often hard
> due to all the other messages.

Currently, only one "terminating connection due to XXXX"
message(WARNING for immediate shutdown, FATAL for fast shutdown) gets
logged in the server logs per backend, so the number of log messages
for each shutdown depends on the number of active backends plus other
bg workers if any. If we don't want to let each active backend to show
up these messages separately, then how about postmaster (as it anyways
knows what are the active backends it currently has) checking if all
the backends have exited properly and showing only one message,
something like "the active backends are terminated due to XXXX"?

Thoughts?

With Regards,
Bharath Rupireddy.
EnterpriseDB: http://www.enterprisedb.com