Add jsonlog log_destination for JSON server logs

From: Sehrope Sarkuni <sehrope(at)jackdb(dot)com>
To: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Cc: Michael Paquier <michael(at)paquier(dot)xyz>, david(at)fetter(dot)org
Subject: Add jsonlog log_destination for JSON server logs
Date: 2021-08-31 15:34:56
Message-ID: CAH7T-aqswBM6JWe4pDehi1uOiufqe06DJWaU5=X7dDLyqUExHg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi,

This patch adds a new log_destination, "jsonlog", that writes log entries
as lines of JSON. It was originally started by David Fetter using
the jsonlog module by Michael Paquier (
https://github.com/michaelpq/pg_plugins/blob/master/jsonlog/jsonlog.c) as a
basis for how to serialize the log messages. Thanks to both of them because
this wouldn't be possible without that starting point.

The first commit splits out the destination in log pipe messages into its
own field. Previously it would piggyback on the "is_last" field. This adds
an int to the message size but makes the rest of the code easier to follow.

The second commit adds a TAP test for log_destination "csvlog". This was
done to both confirm that the previous change didn't break anything and as
a skeleton for the test in the next commit.

The third commit adds the new log_destination "jsonlog". The output format
is one line per entry with the top level output being a JSON object keyed
with the log fields. Newlines in the output fields are escaped as \n so the
output file has exactly one line per log entry. It also includes a new test
for verifying the JSON output with some basic regex checks (similar to the
csvlog test).

Here's a sample of what the log entries look like:

{"timestamp":"2021-08-31 10:15:25.129
EDT","user":"sehrope","dbname":"postgres","pid":12012,"remote_host":"[local]","session_id":"612e397d.2eec","line_num":1,"ps":"idle","session_start":"2021-08-31
10:15:25
EDT","vxid":"3/2","txid":"0","error_severity":"LOG","application_name":"
006_jsonlog.pl","message":"statement: SELECT 1/0"}

It builds and passes "make check-world" on Linux. It also includes code to
handle Windows as well but I have not actually tried building it there.

Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/

Attachment Content-Type Size
v001-0001-Adds-separate-dest-field-to-log-protocol-PipeProtoHe.patch text/x-patch 4.3 KB
v001-0002-Add-TAP-test-for-csvlog.patch text/x-patch 3.5 KB
v001-0003-Add-jsonlog-log_destination-for-JSON-server-logs.patch text/x-patch 31.6 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andres Freund 2021-08-31 15:53:33 Re: prevent immature WAL streaming
Previous Message vignesh C 2021-08-31 15:24:11 Re: Added missing invalidations for all tables publication