Re: Built-in plugin for logical decoding output

On 23/09/17 19:01, Alvaro Hernandez wro> On 23/09/17 18:42, Euler
Taveira wrote:
>> 2017-09-22 19:28 GMT-03:00 Gregory Brail <gregbrail(at)google(dot)com>:
>>> We have been working on a project that makes extensive use of logical
>>> replication for use inside Apigee (which is a very small part of
>>> Google):
>>>
>>> https://github.com/apigee-labs/transicator
>>>
>>> In order to do this, we had to write our own logical replication plugin
>>> because the supplied "test_decoding" plugin from the "contrib"
>>> directory was
>>> hard for us to work with. Primarily:
>>>
>> test_decoding is a proof of concept to illustrate the logical decoding
>> potential. It is not intended for production.
>
>     However, AFAIK, AWS's DMS uses it for production purposes (see
> http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html).
>
>
>> I developed wal2json [1]
>> for general use. It outputs changes in JSON. It was one of the first
>> logical decoding plugins.
>>
>>> 1) It doesn't include all the fields that we need for Transicator (most
>>> importantly we need the LSN and the 64-bit transaction ID).
>>>
>> wal2json includes both.
>>
>>> 2) It outputs a text format that is hard to parse.
>>>
>> There are a lot of JSON parsers.
>>
>>> I imagine that other users of logical decoding are facing similar
>>> problems.
>>>
>>> Would the community support the development of another plugin that is
>>> distributed as part of "contrib" that addresses these issues? I'd be
>>> happy
>>> to submit a patch, or GitHub repo, or whatever works best as an example.
>>> (Also, although Transicator uses protobuf, I'm happy to have it output a
>>> simple binary format as well.)
>>>
>> There was a prior discussion and it was suggestted that we have a
>> ready-for-production plugin in core (besides pgoutput). It was
>> suggested [1] that I submit wal2json for 11. I'm in process to clean
>> up the code and hope to submit it to CF2.

Thanks, I'll be happy to review that.

>
>     I would be happy to see another logical decoding plugin into core
> starting on 11. However, this also poses a bit of a challenge for
> middleware implementors: you need to support one for 9.4-9.5
> (test_decoding), another for 10 (pgoutput) and maybe another for 11
> onwards. The idea of asking users to install a binary plugin is very
> unsexy, so these are the options available.

Well, test_decoding is not meant for production use anyway, no need for
middleware to support it. The pgoutput is primarily used for internal
replication purposes, which is why we need something with more
interoperability in mind in the first place. The new plugin should still
support publications etc though IMHO.

>     However, having said that, and while json is a great output format
> for interoperability, if there's a discussion on which plugin to include
> next, I'd also favor one that has some more compact representation
> format (or that supports several formats, not only json).
>

JSON is indeed great for interoperability, if you want more compact
format, use either pgoutput or write something of your own or do
conversion to something else in your consumer. I don't think postgres
needs to provide 100 different formats out of the box when there is an
API. The JSON output does not have to be extremely chatty either btw.

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services