Re: Built-in plugin for logical decoding output

On 09/25/2017 12:48 PM, Alvaro Hernandez wrote:
>
>
> On 25/09/17 19:39, Petr Jelinek wrote:
>>
>> Well, test_decoding is not meant for production use anyway, no need for
>> middleware to support it. The pgoutput is primarily used for internal
>> replication purposes, which is why we need something with more
>> interoperability in mind in the first place. The new plugin should still
>> support publications etc though IMHO.
>>
>>>      However, having said that, and while json is a great output format
>>> for interoperability, if there's a discussion on which plugin to
>>> include
>>> next, I'd also favor one that has some more compact representation
>>> format (or that supports several formats, not only json).
>>>
>> JSON is indeed great for interoperability, if you want more compact
>> format, use either pgoutput or write something of your own or do
>> conversion to something else in your consumer. I don't think postgres
>> needs to provide 100 different formats out of the box when there is an
>> API. The JSON output does not have to be extremely chatty either btw.
>>
>
>     In my opinion, logical decoding plugins that don't come with core
> are close to worthless (don't get me wrong):
>
> - They very unlikely will be installed in managed environments (an
> area growing significantly).
> - As anything that is not in core, raises concerns by users.
> - Distribution and testing are non-trivial: many OS/archs combinations.
>
>     Given the above, I believe having a general-purpose output plugin
> in-core is critical to the use of logical decoding. As for 9.4-9.6
> there is test_decoding, and given that AWS uses it for production,
> that's kind of fine. For 10 there is at least pgoutput, which could be
> used (even though it was meant for replication). But if a new plugin
> is to be developed for 11+, one really general purpose one, I'd say
> json is not a good choice if it is the only output it would support.
> json is too verbose, and replication, if anything, needs performance
> (it is both network heavy and serialization/deserialization is quite
> expensive). Why not, if one and only one plugin would be developed for
> 11+, general purpose, do something that is, indeed, more general,
> i.e., that supports high-performance scenarios too?
>
>
>   

A general purpose lower bandwidth plugin might one supporting Protocol
Buffers. The downside is that unlike json it's not self-contained, you
need the message definitions to interpret the stream, AIUI.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services