Re: Built-in plugin for logical decoding output

Thanks! I didn't read the version 10 RC docs carefully enough.

I found the code that generates the protocol message (looks like its
replication/proto/logical.c). Are there docs somewhere on the format, or is
it just the code?

Also in lieu of the new snapshot mechanism for logical replication, which
might not work for us, we were using the transaction ID to calculate what
was committed in a client's snapshot and what they need to apply to their
own local database. That relied on the transaction ID, and we wanted to use
a 64-bit ID so that we could handle rollover. We ended up doing this:

https://github.com/apigee-labs/transicator/blob/2d5dc596a5f2f5e13967e0f1943f248d88eac1e7/pgoutput/transicator_output.c#L151

It looks to me like the new stuff only puts a 32-bit "xid" in there. Would
there be a way to include the epoch as well? (And yes, I realize it might
require a more detailed explanation which I'm happy to put together.)

On Fri, Sep 22, 2017 at 4:01 PM, Alvaro Hernandez <aht(at)ongres(dot)com> wrote:

>
>
> On 23/09/17 00:28, Gregory Brail wrote:
>
>> We have been working on a project that makes extensive use of logical
>> replication for use inside Apigee (which is a very small part of Google):
>>
>> https://github.com/apigee-labs/transicator
>>
>> In order to do this, we had to write our own logical replication plugin
>> because the supplied "test_decoding" plugin from the "contrib" directory
>> was hard for us to work with. Primarily:
>>
>> 1) It doesn't include all the fields that we need for Transicator (most
>> importantly we need the LSN and the 64-bit transaction ID).
>> 2) It outputs a text format that is hard to parse.
>>
>> I imagine that other users of logical decoding are facing similar
>> problems.
>>
>> Would the community support the development of another plugin that is
>> distributed as part of "contrib" that addresses these issues? I'd be happy
>> to submit a patch, or GitHub repo, or whatever works best as an example.
>> (Also, although Transicator uses protobuf, I'm happy to have it output a
>> simple binary format as well.)
>>
>> As a side note, doing this would also help making logical decoding a
>> useful feature for customers of Amazon and Google's built-in Postgres
>> hosting options. In those environments, there's no way to add a custom
>> plugin to Postgres, so anything not built in the product tends to be harder
>> for someone to consume.
>>
>> If anyone is interested in looking more:
>>
>> The plugin code is here:
>> https://github.com/apigee-labs/transicator/tree/master/pgoutput
>>
>> and produces output defined by the "ChangePb" structure defined here:
>> https://github.com/apigee-labs/transicator/blob/master/commo
>> n/transicator.proto
>>
>
> How about using pgoutput, which is included by default in PostgreSQL
> 10, as the basis for logical replication?
>
>
> Cheers,
>
> Álvaro
>
>
> --
>
> Alvaro Hernandez
>
>
> -----------
> OnGres
>
>