Re: Streaming solution and v3.1 protocol

On 03.06.2011 19:19, Radosław Smogura wrote:
> Hello,
>
> Sorry for short introduction about this, and plese as far as possible,
> disconnet it from LOBs, as it on top of LOB.
>
> Idea of streaming is to reduce memory copy mainly during receiving and sending
> tuples. Currently receive works as follows
> 1. Read bytes of tuple (allocate x memory).
> 2. Eventually convert it to database encoding.
> 3. Use this data and create datum (which is little changed copy of 1 or 2)
> 4. Streaming will be allowed only in binary mode, and actually stream in/out
> will return binary data.

Hmm, I was thinking that streaming would be a whole new mode, alongside
the current text and binary mode.

> Look for example at execution chain from textrecv.
>
> Idea is to add stream_in, stream_out columns pg_type.
>
> When value is to be serialized the sin/sout is called. Struct must pass len of
> data, and struct of stream (simillar to C FILE*).
>
> Caller should validate if all bytes has been consumed (expose simple methods
> for this)
>
> To implement(code API requirements):
> First stream is buffered socekt reader.
>
> Fast substreams - create fast stream limited to x bytes basing on other stream
>
> Skipping bytes + skipAll()
>
> Stream filtering - do fast (faster will be if conversion will occure in
> buffered chunks) encoding conversion.
>
> Support for direct PG printf() version. Linux has ability to create cookie
> streams and use it with fprintf(), so its greate advantage to format huge
> strings. Other system should buffer output. Problem is if Linux cookie will
> fail will it write something to output? Windows proxy will push value to temp
> buffer.

This is pretty low-level stuff, I think we should focus on the protocol
changes and user-visible libpq API first.

However, we don't want to use anything Linux-specific here, so that
cookie streams are not an option.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com