Re: Refactoring syslogger piping to simplify adding new log destinations

Hi Sehrope,

On 2019-Jul-10, Sehrope Sarkuni wrote:

> While working on adding a new log_destination I noticed that the
> syslogger piping would need to be updated. At the moment both ends
> only handle stderr/csvlog as the pipe message header has a char
> "is_last" that is either t/f (stderr last, stderr partial) or T/F
> (csvlog last, csvlog partial).

I spent some time a couple of weeks ago looking at profiles of the
syslogger code, and my impression is that the current design of using a
pipe to move data from one process to another may benefit from a
complete rewrite; it seems that the kernel overhead of going over the
pipe is significant. (The test case was a couple dozen processes all
generating a thousand of couple-hundred-KB log lines. In perf, the pipe
read/write takes up 99% of the CPU time).

Maybe we can use something like a shared memory queue, working in a
similar way to wal_buffers -- where backends send over the shm queue to
syslogger, and syslogger writes in order to the actual log file. Or
maybe something completely different; I didn't actually prototype
anything, just observed the disaster.

I'm not opposed to your patches, just trying to whet your appetite for
something bigger in the vicinity.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services