Re: Built-in plugin for logical decoding output

On 23/09/17 00:28, Gregory Brail wrote:
> We have been working on a project that makes extensive use of logical
> replication for use inside Apigee (which is a very small part of Google):
>
> https://github.com/apigee-labs/transicator
>
> In order to do this, we had to write our own logical replication
> plugin because the supplied "test_decoding" plugin from the "contrib"
> directory was hard for us to work with. Primarily:
>
> 1) It doesn't include all the fields that we need for Transicator
> (most importantly we need the LSN and the 64-bit transaction ID).
> 2) It outputs a text format that is hard to parse.
>
> I imagine that other users of logical decoding are facing similar
> problems.
>
> Would the community support the development of another plugin that is
> distributed as part of "contrib" that addresses these issues? I'd be
> happy to submit a patch, or GitHub repo, or whatever works best as an
> example. (Also, although Transicator uses protobuf, I'm happy to have
> it output a simple binary format as well.)
>
> As a side note, doing this would also help making logical decoding a
> useful feature for customers of Amazon and Google's built-in Postgres
> hosting options. In those environments, there's no way to add a custom
> plugin to Postgres, so anything not built in the product tends to be
> harder for someone to consume.
>
> If anyone is interested in looking more:
>
> The plugin code is here:
> https://github.com/apigee-labs/transicator/tree/master/pgoutput
>
> and produces output defined by the "ChangePb" structure defined here:
> https://github.com/apigee-labs/transicator/blob/master/common/transicator.proto

    How about using pgoutput, which is included by default in
PostgreSQL 10, as the basis for logical replication?

    Cheers,

    Álvaro

--

Alvaro Hernandez

-----------
OnGres