Built-in plugin for logical decoding output

From: Gregory Brail <gregbrail(at)google(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Built-in plugin for logical decoding output
Date: 2017-09-22 22:28:56
Message-ID: CAFF4x11S20YPemiM5UbM1YjPy5rwF-i+D=vhzw8r8OB_A0dktw@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

We have been working on a project that makes extensive use of logical
replication for use inside Apigee (which is a very small part of Google):

https://github.com/apigee-labs/transicator

In order to do this, we had to write our own logical replication plugin
because the supplied "test_decoding" plugin from the "contrib" directory
was hard for us to work with. Primarily:

1) It doesn't include all the fields that we need for Transicator (most
importantly we need the LSN and the 64-bit transaction ID).
2) It outputs a text format that is hard to parse.

I imagine that other users of logical decoding are facing similar problems.

Would the community support the development of another plugin that is
distributed as part of "contrib" that addresses these issues? I'd be happy
to submit a patch, or GitHub repo, or whatever works best as an example.
(Also, although Transicator uses protobuf, I'm happy to have it output a
simple binary format as well.)

As a side note, doing this would also help making logical decoding a useful
feature for customers of Amazon and Google's built-in Postgres hosting
options. In those environments, there's no way to add a custom plugin to
Postgres, so anything not built in the product tends to be harder for
someone to consume.

If anyone is interested in looking more:

The plugin code is here:
https://github.com/apigee-labs/transicator/tree/master/pgoutput

and produces output defined by the "ChangePb" structure defined here:
https://github.com/apigee-labs/transicator/blob/master/common/transicator.proto

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Alvaro Hernandez 2017-09-22 23:01:13 Re: Built-in plugin for logical decoding output
Previous Message Claudio Freire 2017-09-22 22:12:01 Re: Small improvement to compactify_tuples