Re: Reopen logfile on SIGHUP

From: Alexander Kuzmenkov <a(dot)kuzmenkov(at)postgrespro(dot)ru>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, Andres Freund <andres(at)anarazel(dot)de>, Greg Stark <stark(at)mit(dot)edu>, Anastasia Lubennikova <a(dot)lubennikova(at)postgrespro(dot)ru>, pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>, Grigory Smolkin <g(dot)smolkin(at)postgrespro(dot)ru>
Subject: Re: Reopen logfile on SIGHUP
Date: 2018-04-12 14:23:42
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On 11.04.2018 00:00, Tom Lane wrote:
> So we need a mechanism that's narrowly targeted
> to reopening the logfile, without SIGHUP'ing the entire database.

We can send SIGUSR1 to the syslogger process. To make its pid easier to
find out, it can be published in "$PGDATA/", as
suggested by Grigory. The attached patch does this. It also adds a brief
description of how to use this with logrotate.

> Point 2: Depending on how you've got the log filenames configured,
> setting rotation_requested may result in a change in log filename

If logrotate only needs the file to be reopened, syslogger's rotation
does just than when using a static log file name. I imagine logrotate
can be configured to do something useful with changing file names, too.
It is a matter of keeping the configuration of syslogger and logrotate

> BTW, another thing that needs to be considered is the interaction with
> rotation_disabled. Right now we automatically drop that on SIGHUP, but
> I'm unclear on whether it should be different for logrotate requests.

The SIGUSR1 path is supposed to be used by automated tools. In a sense,
it is an automatic rotation, the difference being that it originates
from an external tool and not from syslogger itself. So, it sounds
plausible that the rotation request shouldn't touch the
rotation_disabled flag, and should be disabled by it, just like the
automatic rotation.

Still, this leads us to a scenario where we can lose logs:
1. postgres is configured to use a static file name. logrotate is
configured to move the file, send SIGUSR1 to postgres syslogger, gzip
the file and delete it.
2. logrotate starts the rotation. It moves the file and signals postgres
to reopen it.
3. postgres fails to reopen the file because there are too many files
open (ENFILE/EMFILE), which is a normal occurrence on heavily loaded
systems. Or it doesn't open the new file because the rotation_disable
flag is set. It continues logging to the old file.
4. logrotate has no way to detect this failure, so it gzips the file and
unlinks it.
5. postgres continues writing to the now unlinked file, and we lose an
arbitrary amount of logs until the next successful rotation.

With dynamic file names, logrotate can be told to skip open files, so
that it doesn't touch our log file if we haven't switched to the new
one. With a static file name, the log file is always open, so this
method doesn't work. I'm not sure how to make this work reliably.

Alexander Kuzmenkov
Postgres Professional:
The Russian Postgres Company

Attachment Content-Type Size
publish-syslogger-pid-v1.patch text/x-patch 3.9 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Teodor Sigaev 2018-04-12 14:27:20 Re: Partitioned tables and covering indexes
Previous Message Robert Haas 2018-04-12 14:21:34 Re: Commit 4dba331cb3 broke ATTACH PARTITION behaviour.