Re: Built-in plugin for logical decoding output

On 26/09/17 10:03, Craig Ringer wrote:
> On 26 September 2017 at 14:08, Alvaro Hernandez <aht(at)ongres(dot)com
> <mailto:aht(at)ongres(dot)com>> wrote:
>
>
>
>     OK, let me try to do that. I believe data integration is a
> priority.
>
>
> Definitely agree so far.
>
> - If you want to develop your own output plugin, then your market
> is reduced as you have to exclude all the managed solutions
> (until, and only if, you would convince them to include your
> plugin... highly unlikely, very difficult). And probably another %
> of enterprise environments which will hesitate to link your own
> plugin to the running production database. Last but not least, you
> need to compile and test (and testing this is far from easy) on
> multiple OSes/architectures.
>
>
> Right. You probably don't want one output plugin per application.
>
> This doesn't mean it has to be in-core, just flexible and share-able
> by many, and adopted by cloud vendors. Like say PostGIS.

    That would be nice. But this is chicken-and-egg: an out-of-core
plugin won't probably be used by many if applications like the ones I
was mentioning if they do not exist. And developing such an application
is so much less interesting if a significant part of your market is
excluded from your app.

    However, it could work the other way around: a sufficiently good
enough in-core base plugin could foster applications/ecosystem, which
once adopted by users could push much more easily for other more
advanced out-of-core plugins, that would be more easily accepted by
pressure as those tools would already be with significant traction. But
I don't see it the other way around.

>
> - If you stick to in-core plugins, then you need to support at
> least three different output formats if you want to support 9.4+:
> test_decoding (and pray it works!), pgoutput, and the "new"
> in-core plugin that was proposed at the beginning of this thread,
> if that would see the light.
>
>
> The only practical way will IMO be to have whatever new plugin it also
> have an out-of-core version maintained for older Pg versions, where it
> can be installed.
>
> But only in-core plugins help for general-purpose solutions.
>
>
> I still don't agree there. If there's enough need/interest/adoption
> you can get cloud vendors on board, they'll feel the customer
> pressure. It's not our job to create that pressure and do their work
> for them.

    Don't want to get into a loop, but as I said before it's
chicken-and-egg. But nobody is asking core to do their work. As much as
I love it, I think logical decoding is a bit half-baked until there is a
single, quality, in-core plugin, as it discourages its usage, because of
the reasons I stated.

>
> I see nothing wrong with a plugin starting off out of core and being
> adopted+adapted later, assuming it's done well.
>
> That said, I'm all in favour of a generic json output plugin that
> shares infrastructure with logical replication, so people who are on
> inflexible environments have a fallback option. I just don't care to
> write it.

    That's better than nothing. But as much as interoperable json may
be, people still need to talk the (binary) replication protocol to use
it. So once you talk binary protocol, why not talk binary also for the
output plugin and have a much more efficient output? Again, nothing
against json, but if a new plugin would be included in-core, I'd say
json + binary also. Or just document pgoutput, as it could be good enough.

    Cheers,

    Álvaro

--

Alvaro Hernandez

-----------
OnGres