Re: Built-in plugin for logical decoding output

On 23/09/17 18:42, Euler Taveira wrote:
> 2017-09-22 19:28 GMT-03:00 Gregory Brail <gregbrail(at)google(dot)com>:
>> We have been working on a project that makes extensive use of logical
>> replication for use inside Apigee (which is a very small part of Google):
>>
>> https://github.com/apigee-labs/transicator
>>
>> In order to do this, we had to write our own logical replication plugin
>> because the supplied "test_decoding" plugin from the "contrib" directory was
>> hard for us to work with. Primarily:
>>
> test_decoding is a proof of concept to illustrate the logical decoding
> potential. It is not intended for production.

    However, AFAIK, AWS's DMS uses it for production purposes (see
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html).

> I developed wal2json [1]
> for general use. It outputs changes in JSON. It was one of the first
> logical decoding plugins.
>
>> 1) It doesn't include all the fields that we need for Transicator (most
>> importantly we need the LSN and the 64-bit transaction ID).
>>
> wal2json includes both.
>
>> 2) It outputs a text format that is hard to parse.
>>
> There are a lot of JSON parsers.
>
>> I imagine that other users of logical decoding are facing similar problems.
>>
>> Would the community support the development of another plugin that is
>> distributed as part of "contrib" that addresses these issues? I'd be happy
>> to submit a patch, or GitHub repo, or whatever works best as an example.
>> (Also, although Transicator uses protobuf, I'm happy to have it output a
>> simple binary format as well.)
>>
> There was a prior discussion and it was suggestted that we have a
> ready-for-production plugin in core (besides pgoutput). It was
> suggested [1] that I submit wal2json for 11. I'm in process to clean
> up the code and hope to submit it to CF2.

    I would be happy to see another logical decoding plugin into core
starting on 11. However, this also poses a bit of a challenge for
middleware implementors: you need to support one for 9.4-9.5
(test_decoding), another for 10 (pgoutput) and maybe another for 11
onwards. The idea of asking users to install a binary plugin is very
unsexy, so these are the options available.

    However, having said that, and while json is a great output format
for interoperability, if there's a discussion on which plugin to include
next, I'd also favor one that has some more compact representation
format (or that supports several formats, not only json).

    Regards,

    Álvaro

--

Alvaro Hernandez

-----------
OnGres