Re: [PROTOCOL TODO] Permit streaming of unknown-length lob/clob (bytea, text, etc)

Craig Ringer <craig(at)2ndquadrant(dot)com> writes:
> Currently the client must know the size of a large lob/clob field, like
> a 'bytea' or 'text' field, in order to send it to the server. This can
> force the client to buffer all the data before sending it to the server.

> It would be helpful if the v4 protocol permitted the client to specify
> the field length as unknown / TBD, then stream data until an end marker
> is read. Some encoding would be required for binary data to ensure that
> occurrences of the end marker in the streamed data were properly
> handled, but there are many well established schemes for doing this.

I think this is pretty much a non-starter as stated, because the v3
protocol requires all messages to have a preceding length word. That's
not very negotiable.

What's already on the TODO list is to allow large field values to be sent
or received in segments, perhaps with a cursor-like arrangement. You can
do that today for blobs, but not for oversize regular table fields.

Of course, considering that the maximum practical size of a regular field
is probably in the dozens of megabytes, and that RAM is getting cheaper
all the time, it's not clear that it's all that much of a hardship for
clients to buffer the whole thing. If we've not gotten around to this
in the last dozen years, it's unlikely we'll get to it in the future
either ...

regards, tom lane