Re: Streaming solution and v3.1 protocol

On Fri, Jun 3, 2011 at 12:04 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> On 03.06.2011 19:19, Radosław Smogura wrote:
>>
>> Hello,
>>
>> Sorry for short introduction about this, and plese as far as possible,
>> disconnet it from LOBs, as it on top of LOB.
>>
>> Idea of streaming is to reduce memory copy mainly during receiving and
>> sending
>> tuples. Currently receive works as follows
>> 1. Read bytes of tuple (allocate x memory).
>> 2. Eventually convert it to database encoding.
>> 3. Use this data and create datum (which is little changed copy of 1 or 2)
>> 4. Streaming will be allowed only in binary mode, and actually stream
>> in/out
>> will return binary data.
>
> Hmm, I was thinking that streaming would be a whole new mode, alongside the
> current text and binary mode.
>
>> Look for example at execution chain from textrecv.
>>
>> Idea is to add stream_in, stream_out columns pg_type.
>>
>> When value is to be serialized the sin/sout is called. Struct must pass
>> len of
>> data, and struct of stream (simillar to C FILE*).
>>
>> Caller should validate if all bytes has been consumed (expose simple
>> methods
>> for this)
>>
>> To implement(code API requirements):
>> First stream is buffered socekt reader.
>>
>> Fast substreams - create fast stream limited to x bytes basing on other
>> stream
>>
>> Skipping bytes + skipAll()
>>
>> Stream filtering - do fast (faster will be if conversion will occure in
>> buffered chunks) encoding conversion.
>>
>> Support for direct PG printf() version. Linux has ability to create cookie
>> streams and use it with fprintf(), so its greate advantage to format huge
>> strings. Other system should buffer output. Problem is if Linux cookie
>> will
>> fail will it write something to output? Windows proxy will push value to
>> temp
>> buffer.
>
> This is pretty low-level stuff, I think we should focus on the protocol
> changes and user-visible libpq API first.

+1. in particular, I'd like to see the libpq api changes.

merlin