Re: Reopen logfile on SIGHUP

On 11.04.2018 00:00, Tom Lane wrote:
> So we need a mechanism that's narrowly targeted
> to reopening the logfile, without SIGHUP'ing the entire database.

We can send SIGUSR1 to the syslogger process. To make its pid easier to
find out, it can be published in "$PGDATA/logging_collector.pid", as
suggested by Grigory. The attached patch does this. It also adds a brief
description of how to use this with logrotate.

> Point 2: Depending on how you've got the log filenames configured,
> setting rotation_requested may result in a change in log filename

If logrotate only needs the file to be reopened, syslogger's rotation
does just than when using a static log file name. I imagine logrotate
can be configured to do something useful with changing file names, too.
It is a matter of keeping the configuration of syslogger and logrotate
consistent.

> BTW, another thing that needs to be considered is the interaction with
> rotation_disabled. Right now we automatically drop that on SIGHUP, but
> I'm unclear on whether it should be different for logrotate requests.

The SIGUSR1 path is supposed to be used by automated tools. In a sense,
it is an automatic rotation, the difference being that it originates
from an external tool and not from syslogger itself. So, it sounds
plausible that the rotation request shouldn't touch the
rotation_disabled flag, and should be disabled by it, just like the
automatic rotation.

Still, this leads us to a scenario where we can lose logs:
1. postgres is configured to use a static file name. logrotate is
configured to move the file, send SIGUSR1 to postgres syslogger, gzip
the file and delete it.
2. logrotate starts the rotation. It moves the file and signals postgres
to reopen it.
3. postgres fails to reopen the file because there are too many files
open (ENFILE/EMFILE), which is a normal occurrence on heavily loaded
systems. Or it doesn't open the new file because the rotation_disable
flag is set. It continues logging to the old file.
4. logrotate has no way to detect this failure, so it gzips the file and
unlinks it.
5. postgres continues writing to the now unlinked file, and we lose an
arbitrary amount of logs until the next successful rotation.

With dynamic file names, logrotate can be told to skip open files, so
that it doesn't touch our log file if we haven't switched to the new
one. With a static file name, the log file is always open, so this
method doesn't work. I'm not sure how to make this work reliably.

--
Alexander Kuzmenkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company