Re: Add minor version to v3 protocol to allow changes without breaking backwards compatibility

On 11/30/2011 06:52 PM, Merlin Moncure wrote:
> On Mon, Nov 28, 2011 at 9:18 AM, Mikko Tiihonen
> <mikko(dot)tiihonen(at)nitorcreations(dot)com> wrote:
>> Hi,
>>
>> As discussed few days ago it would be beneficial if we could change the v3
>> backend<->client protocol without breaking backwards compatibility.
>>
>> Here is a working patch that exports a supported_binary_minor constant and a
>> binary_minor session variable and a that can be used by clients to enable
>> newer features.
>>
>> I also added an example usage where the array encoding for constant size
>> elements is optimized if binary_minor version is new enough.
>>
>> I have coded the client side support for binary_minor for jdbc driver and
>> tested that it worked. But lets first discuss if this is an acceptable path
>> forward.
>
> Regarding your TODO in the code comments about interactions with
> replication: I think it should be removed. WAL streaming depends on
> more things being identical than what is guaranteed here which is
> basically the protocol + data formats.

OK. I'll remove the comments about replication.

> I think all references to
> 'protocol' should be removed; Binary wire formats != protocol: the
> protocol could bump to v4 but the wire formats (by happenstance) could
> all still continue to work -- therefore the version isn't minor (it's
> not dependent on protocol version but only on itself). Therefore,
> don't much like the name 'supported_binary_minor'. How about
> binary_format_version?

I was thinking that it would be possible use the new minor version to
introduce also new protocol messages such as streaming of large data
into database without knowing it's size beforehand.

> Also, is a non granular approach really buying us anything here? ISTM
> *something* is likely to change format on most releases of the server
> so I'm wondering what's the point (if you don't happen to be on the
> same x.x release of the server, you might as well assume to not match
> or at least 'go on your own risk). The value added to the client
> version query is quite low.

You have a very good point about changes in every new postgres
version. Either text or the binary encoding is likely to change for
some types in each new release.

There needs to be decision on official policy about breaking backwards
compatibility of postgresql clients. Is it:

A) Every x.y postgres release is free to break any parts of the
* protocol
* text encoding
* binary protocol
as long as it is documented
-> all client libraries should be coded so that they refuse to
support version x.y+1 or newer (they might have a option to
override this but there are no guarantees that it would work)
Good: no random bugs when using old client library
Bad: initial complaints from users before they understand that
this is the best option for everyone

B) Every x.y postgres release must guarantee that no client visible
parts of protocol, text encoding or binary encoding will change
from previous release in the v3 protocol. If any changes are
needed then a new protocol version must be created.
-> very high barrier for new features
Good: can upgrade server without updating clients
Bad: new features are only introduced very rarely after enough
have accumulated
Bad: many feature/change patches will rot while waiting for the
upcoming new version

C) There is effort to try avoiding incompatible changes. Some
changes are blocked because it is detected that they can break
backwards compatibility while others are let through (often with
some compatibility option on server side to fall back to
previous functionality (f.ex. bytea hex encoding).
-> As far as I understand this is the current situation.
Good: has worked so far
Bad: accumulates compatibility flags in the server

D) My proposed compromise where there is one minor_version setting
and code in server to support all different minor versions.
The client requests the minor version so that all releases
default to backwards compatible version.
When there combinations starts to create too much maintenance
overhead a new clean v4 version of protocol is specified.
Good: Keeps full backwards compatibility
Good: Allows introducing changes at any time
Bad: Accumulates conditional code to server and clients until
new version of protocol is released

I'd like the official policy to be A, otherwise I'll push for D.

-Mikko