Re: Add minor version to v3 protocol to allow changes without breaking backwards compatibility

On Thu, Dec 1, 2011 at 8:42 AM, Mikko Tiihonen
<mikko(dot)tiihonen(at)nitorcreations(dot)com> wrote:
> On 11/30/2011 06:52 PM, Merlin Moncure wrote:
>>
>> On Mon, Nov 28, 2011 at 9:18 AM, Mikko Tiihonen
>> <mikko(dot)tiihonen(at)nitorcreations(dot)com>  wrote:
>>>
>>> Hi,
>>>
>>> As discussed few days ago it would be beneficial if we could change the
>>> v3
>>> backend<->client protocol without breaking backwards compatibility.
>>>
>>> Here is a working patch that exports a supported_binary_minor constant
>>> and a
>>> binary_minor session variable and a that can be used by clients to enable
>>> newer features.
>>>
>>> I also added an example usage where the array encoding for constant size
>>> elements is optimized if binary_minor version is new enough.
>>>
>>> I have coded the client side support for binary_minor for jdbc driver and
>>> tested that it worked. But lets first discuss if this is an acceptable
>>> path
>>> forward.
>>
>>
>> Regarding your TODO in the code comments about interactions with
>> replication:  I think it should be removed.  WAL streaming depends on
>> more things being identical than what is guaranteed here which is
>> basically the protocol + data formats.
>
>
> OK. I'll remove the comments about replication.
>
>
>> I think all references to
>> 'protocol' should be removed;  Binary wire formats != protocol: the
>> protocol could bump to v4 but the wire formats (by happenstance) could
>> all still continue to work -- therefore the version isn't minor (it's
>> not dependent on protocol version but only on itself).  Therefore,
>> don't much like the name 'supported_binary_minor'.  How about
>> binary_format_version?
>
>
> I was thinking that it would be possible use the new minor version to
> introduce also new protocol messages such as streaming of large data
> into database without knowing it's size beforehand.
>
>
>> Also, is a non granular approach really buying us anything here?  ISTM
>> *something* is likely to change format on most releases of the server
>> so I'm wondering what's the point (if you don't happen to be on the
>> same x.x release of the server, you might as well assume to not match
>> or at least 'go on your own risk). The value added to the client
>> version query is quite low.
>
>
> You have a very good point about changes in every new postgres
> version. Either text or the binary encoding is likely to change for
> some types in each new release.
>
> There needs to be decision on official policy about breaking backwards
> compatibility of postgresql clients. Is it:
>
> A) Every x.y postgres release is free to break any parts of the
>   * protocol
>   * text encoding
>   * binary protocol
>   as long as it is documented
>   -> all client libraries should be coded so that they refuse to
>      support version x.y+1 or newer (they might have a option to
>      override this but there are no guarantees that it would work)
>   Good: no random bugs when using old client library
>   Bad: initial complaints from users before they understand that
>        this is the best option for everyone
>
> B) Every x.y postgres release must guarantee that no client visible
>   parts of protocol, text encoding or binary encoding will change
>   from previous release in the v3 protocol. If any changes are
>   needed then a new protocol version must be created.
>   -> very high barrier for new features
>   Good: can upgrade server without updating clients
>   Bad: new features are only introduced very rarely after enough
>        have accumulated
>   Bad: many feature/change patches will rot while waiting for the
>        upcoming new version
>
> C) There is effort to try avoiding incompatible changes. Some
>   changes are blocked because it is detected that they can break
>   backwards compatibility while others are let through (often with
>   some compatibility option on server side to fall back to
>   previous functionality (f.ex. bytea hex encoding).
>   -> As far as I understand this is the current situation.
>   Good: has worked so far
>   Bad: accumulates compatibility flags in the server
>
> D) My proposed compromise where there is one minor_version setting
>   and code in server to support all different minor versions.
>   The client requests the minor version so that all releases
>   default to backwards compatible version.
>   When there combinations starts to create too much maintenance
>   overhead a new clean v4 version of protocol is specified.
>   Good: Keeps full backwards compatibility
>   Good: Allows introducing changes at any time
>   Bad: Accumulates conditional code to server and clients until
>        new version of protocol is released
>
>
> I'd like the official policy to be A, otherwise I'll push for D.

In issue A, you mentioned that client libraries should refuse to
support version x+1 or newer. If by client libraries, you mean
'libpq', then this is absolutely not going to fly -- libpq has no such
restriction now and adding it isn't doing users any favors IMO. The
problem is not libpq/jdbc etc, but application code. I'll say again,
wire format compatibility issues are fundamentally different from the
protocol issues; decades of user application code are involved and
half measures are simply not going to work. The typical scenario is
that some hand written C application utilizes the binary wire format
for some reason and the database goes EOL and gets upgraded, possibly
many years after the application was written. The minor version check
provides zero help in dealing with this problem other then to tell you
something you already knew was going to be an issue, which is really
no help at all. Where a minor version check might find some limited
use is when reading data produced by COPY BINARY, but that's not a
major problem as I see it today since isn't really a huge stumbling
block for users since user application code rarely consumes that
format.

Something that definitely would help binary format consumers while
maintaining the status quo would be to simply document all the wire
formats, or at least all the changes to those formats. That way, at
least someone considering an upgrade would have a handy reference to
check to determine the scope and impact of the changes. As far as I'm
concerned, this puts the text and binary formats at parity in terms of
standard of support to the user.

If you want something richer than a documentation style approach, then
I would be more in favor of something that would actually solve
compatibility issues from the user's point of view. This means
handling wire formats in the client library such that client
application authors are protected from wire format (either text or
binary) changes. Yes, this means we would permanently have to track
old formats forever just like we have to keep the old protocol code
laying around (both in the client *and the server*). Unfortunately,
this only partially helps jdbc users, since the jdbc doesn't wrap
libpq. If the jdbc authors want to get serious about binary support
of fancy backend structures like arrays then they would have to get
serious about client insulation as well. This is, by the way, a noble
goal. The Postgres type system is incredibly rich but many
developers, especially non C coders, are unable to fully take
advantage of that because of poor support on the client side.

merlin