Re: Similar to csvlog but not really, json logs?

Tom,

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> Stephen Frost <sfrost(at)snowman(dot)net> writes:
> > The flip side is that there are absolutely production cases where what
> > we output is either too little or too much- being able to control that
> > and then have the (filtered) result in JSON would be more-or-less
> > exactly what a client of ours is looking for.
>
> > To try to clarify that a bit, as it comes across as rather opaque even
> > on my re-reading, consider a case where you can't have the
> > "credit_card_number" field ever exported to an audit or log file, but
> > you're required to log all other changes to a table. Then consider that
> > such a situation extends to individual INSERT or UPDATE commands- you
> > need the command logged, but you can't have the contents of that column
> > in the log file.
>
> Hmm ... that's a lovely use-case, but somehow I don't find "let's output
> in JSON" to be a credible solution. Not only are you paying extra output
> log volume to do that, but you are also supposing that some downstream
> process is going to parse the JSON, remove some data (which JSON-izing
> the log format didn't especially help with, be honest), and re-emit JSON
> afterwards. There's no way that that scales to any production situation,
> even if you had parallel log-filtering processes which you won't.

That's not what I was trying to suggest, though I understand how I
wasn't clear. There's no downstream process to parse and filter the
JSON, in an ideal situation.

Consider an audit system where which columns end up in the audit log are
controlled by issuing ALTER TABLE .. ALTER COLUMN type statements.
Where JSON plays into this is that we can't depend on the table
structure to tell us what the columns are in an audit log any longer
and therefore that information needs to be included- and JSON happens to
conveniently do that.

As for parallel logging- that's a different subject but one which a lot
of people are very interested in (I expect Nasby would be happy to
comment on that subject, as he and I discussed it a number of years ago
at PGCon, since their systems are one of those where you simply can't
turn on query logging or the system will essentially stop).

> I am interested in thinking about your scenario; I just don't think
> that JSON output format is any real part of the answer.

I'm certainly open to other suggestions. JSON was suggested by the
client as a possible approach, but better ideas are always welcome.

Thanks,

Stephen