Re: libpq and Binary Data Formats

On 6/4/07, Wilhansen Li <willi(dot)t1(at)gmail(dot)com> wrote:
> First of all, apologies if this was not meant to be a feedback/wishlist
> mailing list.
>
> Binary formats in libpq has been (probably) a long issue (refer to the
> listings below) and I want to express my hope that the next
> revision of PostgreSQL would have better support for binary data types in
> libpq. I am in no doubt that those binary vs. text debates sprouted because
> of PostgreSQL's (or rather libpq's) ambiguity when it comes to binary data
> support. One instance is the documentation itself: it didn't really say
> (correct me if I'm wrong) that binary data is poorly/not supported and that
> textual data is preferred. Moreover, those ambiguities are only cleared up
> in mailing lists/irc/forums which make it seem that the arguments for text
> data is just an excuse to not have proper support for binary data ( e.x.
> C:"Elephant doesn't support Hammer!" P: "You don't really need Hammer (we
> don't support it yet), you can do it with Screwdriver."). This is not meant
> to be a binary vs. text post so I'll reserve my comments for them.
> Nevertheless, they each have their own advantages and disadvantages
> especially when it comes to strongly typed languages that neither shouldn't
> be ignored.
>
> I am well-aware of the problems associated with binary formats and
> backward/forward compatibility:
> http://archives.postgresql.org/pgsql-hackers/1999-08/msg00374.php
> but nevertheless, that shouldn't stop PostgreSQL/libpq's
> hardworking developers from coming up with a solution. The
> earling link showed the interest of using CORBA to handle PostgreSQL objects
> but I belive that it's an overkill and would like to propose using ASN.1
> instead. However, what's important is not really the binary/text
> representation. If we look again the the list below, not everyone need
> binary formats just for speed and efficiency, rather, they need it to be
> able to easily manipulate data. In other words, the interfaces to extract
> data is also important.

Personally, I wouldn't mind seeing the libpq API extended to support
arrays and record structures. PostgreSQL 8.3 is bringing arrays of
composite types and the lack of client side support of these
structures is becoming increasingly glaring. If set up with
text/binary switch, this would deal with at least part of your
objections.

I think most people here would agree that certain aspects of the
documentation of binary formats are a bit weak and could use
improvement (although, it's possible that certain formats were
deliberately not documented because they may change). A classy move
would be to make specific suggestions in -docs and produce a patch.

ISTM to me that many if not most people who are looking at binary
interfaces to the database are doing it for the wrong reasons and you
should consider that when reviewing historical discussions :-). Also,
dealing with large bytea types in the databases which is probably the
most common use case, is pretty well covered in libpq documentation
IMO.

merlin