Re: Binary support for pgoutput plugin

Hi,

On Wed, 5 Jun 2019 at 12:01, Andres Freund <andres(at)anarazel(dot)de> wrote:

> Hi
>
> On June 5, 2019 8:51:10 AM PDT, Dave Cramer <davecramer(at)gmail(dot)com> wrote:
> >On Wed, 5 Jun 2019 at 07:21, Dave Cramer <davecramer(at)gmail(dot)com> wrote:
> >
> >> Hi,
> >>
> >>
> >> On Wed, 5 Jun 2019 at 07:18, Petr Jelinek
> ><petr(dot)jelinek(at)2ndquadrant(dot)com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> On 05/06/2019 00:08, Andres Freund wrote:
> >>> > Hi,
> >>> >
> >>> > On 2019-06-05 00:05:02 +0200, David Fetter wrote:
> >>> >> Would it make sense to work toward a binary format that's not
> >>> >> architecture-specific? I recall from COPY that our binary format
> >is
> >>> >> not standardized across, for example, big- and little-endian
> >machines.
> >>> >
> >>> > I think you recall wrongly. It's obviously possible that we have
> >bugs
> >>> > around this, but output/input routines are supposed to handle a
> >>> > endianess independent format. That usually means that you have to
> >do
> >>> > endianess conversions, but that doesn't make it non-standardized.
> >>> >
> >>>
> >>> Yeah, there are really 3 formats of data we have, text protocol,
> >binary
> >>> network protocol and internal on disk format. The internal on disk
> >>> format will not work across big/little-endian but network binary
> >>> protocol will.
> >>>
> >>> FWIW I don't think the code for binary format was included in
> >original
> >>> logical replication patch (I really tried to keep it as minimal as
> >>> possible), but the code and protocol is pretty much ready for adding
> >that.
> >>>
> >> Yes, I looked through the public history and could not find it.
> >Thanks for
> >> confirming.
> >>
> >>>
> >>> That said, pglogical has code which handles this (I guess Andres
> >means
> >>> that by predecessor of pgoutput) so if you look for example at the
> >>> write_tuple/read_tuple/decide_datum_transfer in
> >>>
> >>>
> >
> https://github.com/2ndQuadrant/pglogical/blob/REL2_x_STABLE/pglogical_proto_native.c
> >>> that can help you give some ideas on how to approach this.
> >>>
> >>
> >> Thanks for the tip!
> >>
> >
> >Looking at:
> >
> https://github.com/postgres/postgres/blob/8255c7a5eeba8f1a38b7a431c04909bde4f5e67d/src/backend/replication/pgoutput/pgoutput.c#L163
> >
> >this seems completely ignored. What was the intent?
>
> That's about the output of the plugin, not the datatypes. And independent
> of text/binary output, the protocol contains non-printable chars.
>
> Andres
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

So one of the things they would like added is to get not null information
in the schema record. This is so they can mark the field Optional in Java.
I presume this would also have some uses in other languages. As I
understand it this would require a protocol bump. If this were to be
accepted are there any outstanding asks that would useful to add if we were
going to bump the protocol?

Dave