Re: Auditing extension for PostgreSQL (Take 2)

From: Bruce Momjian <bruce(at)momjian(dot)us>
To: Stephen Frost <sfrost(at)snowman(dot)net>
Cc: Peter Eisentraut <peter_e(at)gmx(dot)net>, Fujii Masao <masao(dot)fujii(at)gmail(dot)com>, Sawada Masahiko <sawada(dot)mshk(at)gmail(dot)com>, David Steele <david(at)pgmasters(dot)net>, Simon Riggs <simon(at)2ndquadrant(dot)com>, Tatsuo Ishii <ishii(at)postgresql(dot)org>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>, Abhijit Menon-Sen <ams(at)2ndquadrant(dot)com>
Subject: Re: Auditing extension for PostgreSQL (Take 2)
Date: 2015-05-09 22:42:25
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Thu, May 7, 2015 at 03:41:13PM -0400, Stephen Frost wrote:
> Bruce,
> > What is our history of doing things in contrib because we are not sure
> > what we want, then moving it into core? My general recollection is that
> > there is usually something in the contrib version we don't want to add
> > to core and people are locked into the contrib API, so we are left
> > supporting it, e.g. xml2, though you could argue that auditing doesn't
> > have application lock-in and xml2 was tied to an external library
> > feature.
> That's exactly the argument that I'd make there. My recollection is
> that we did move pieces of hstore and have moved pieces of other contrib
> modules into core; perhaps we've not yet had a case where we've
> completely pulled one in, but given the relatively low level of
> dependency associated with pgAudit, I'm certainly hopeful that we'll be
> able to here. Lack of history which could be pointed to that's exactly
> what I'm suggesting here doesn't seem like a reason to not move forward
> here though; the concept of having a capability initially in contrib and
> then bringing it into core has certainly been discussed a number of
> times on other threads and generally makes sense, at least to me,
> especially when there's little API associated with the extension.

OK, I am just asking so we remember this can go badly, not that it will.

> > I guess the over-arching question is whether we have to put this into
> > contrib so we can get feedback and change the API, or whether using from
> > PGXN or incrementally adding it to core is the right approach.
> I'm surprised to hear this question of if we "have to" do X, Y, or Z.
> pgAudit brings a fantastic capability to PostgreSQL which users have
> been asking to have for many years and is a feature we should be itching
> to have included. That we can then take it and incrementally add it to
> core, to leverage things which are only available in core (as discussed
> last summer, including grammar and relation metadata), looks to me like
> a great direction to go in and one which we could use over and over to
> bring new features and capabilities to PG.
> Lack of auditing is one of the capabilities that users coming from other
> large RDBMS's see as preventing their ability to migrate to PostgreSQL.
> Other databases (open and closed source) have it and have had it for
> years and it's a serious shortcoming of ours that makes users either
> stick with their existing vendor or look to other closed-source or even
> open-source solutions.

Yes, more extensive auditing is definitely needed.

> > > involved in this discussion will be also. Additionally, as discussed
> > > last summer, we can provide a migration path (which does not need to be
> > > automated or even feature compatible) from pgAudit to an in-core
> > > solution and then sunset pgAudit.
> >
> > Uh, that usually ends badly too.
> I'm confused by this, as it was the result of our discussion and your
> suggestion from last summer: 20140730192136(dot)GM2791(at)momjian(dot)us
> I certainly hope that hasn't substantially changed as that entire
> discussion is why we're even able to have this discussion about
> including pgAudit now. I was very much on-board with trying to work on
> an in-core solution until that thread convinced me that the upgrade
> concerns which I was worried about wouldn't be an issue for inclusion of
> an extension to provide the capability.

I had forgotten about that. Yes, pg_upgrade can easily do this.

> > > Building an in-core solution, in my estimation at least, is going to
> > > require at least a couple of release cycles and having the feedback from
> > > users of pgAudit will be very valuable to building a good solution, but
> > > I don't believe we'll get that feedback without including it.
> >
> > See above --- is it jump through the user hoops and only then they will
> > use it and give us feedback? How motivated can they be if they can't
> > use the PGXN version?
> Why wouldn't we want to include this capability in PG? I also addressed
> the "why not PGXN" above. It it not a lack of motivation but the entire
> intent and design of the PGXN system which precludes most large
> organizations from using it, particularly for sensitive requirements
> such as auditing.

So they trust the Postgres, but not the PGXN authors --- I guess legally
that makes sense.

> > The bottom line is that for the _years_ we ship pg_audit in /contrib, we
> > will have some logging stuff in postgresql.conf and some in
> > contrib/pg_audit and that distinction is going to look quite odd. To
> > the extent you incrementally add to core, you will have duplicate
> > functionality in both places.
> That's entirely correct, of course, but I'm not seeing it as an issue.
> I'm certainly prepared to support shipping pgAudit in contrib, as are
> others based on how this feature has been developed, for the years that
> we'll have 9.5, 9.6 (or 10.0, etc) supported- and that's also another
> reason why users will use it when they wouldn't use something on PGXN.
> Further, I look forward to working incrementally to bring similar
> capability into core, but I suspect those increments will largely be in
> the infrastructure until we reach the point where we're able to provide
> the user-facing bits, which is quite likely to go in all at once and
> allow us a clear upgrade path from one to the other. Perhaps that's
> optimistic, but we do tend to try and bring things in as whole
> capabilities rather than bits and pieces and I don't expect us to need
> to do it differently here.

OK, I just felt I had to ask those questions so we know where the
pitfalls are --- over-optimism always sets of alarms for me.

Bruce Momjian <bruce(at)momjian(dot)us>

+ Everyone has their own god. +

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Andres Freund 2015-05-09 22:44:54 Re: Manipulating complex types as non-contiguous structures in-memory
Previous Message Tom Lane 2015-05-09 21:40:50 Re: initdb -S and tablespaces