Re: Support of partial decompression for datums

On Fri, 4 Dec 2015 22:13:58 +0900
Michael Paquier <michael(dot)paquier(at)gmail(dot)com> wrote:

> On Fri, Dec 4, 2015 at 9:47 PM, Ildus Kurbangaliev
> <i(dot)kurbangaliev(at)postgrespro(dot)ru> wrote:
> > Attached patch adds support of partial decompression for datums.
> > It will be useful in many cases when extracting part of data is
> > enough for big varlena structures.
> >
> > It is especially useful for expanded datums, because it provides
> > storage for partial results.
> >
> > I have another patch, which removes the 1 Mb limit on tsvector using
> > this feature.
>
> -1 for changing the shape of pglz_decompress directly and particularly
> use metadata in it. The current format of those routines is close to
> what lz4 offers in terms of compression and decompression of a string,
> let's not break that we had a time hard enough in 9.5 cycle to get
> something clean.

Metadata is not used for current code, only for case with partial
decompression.

>
> By the way, why don't you compress the multiple chunks and store the
> related metadata at a higher level? There is no need to put that in
> pglz itself.

Yes, but this idea with chunks means that you are creating a whole
new structure, but it can't be used if you want to optimize current
structures. For example you can't just change arrays, but I think there
are places where the partial decompression can be used.

--
Ildus Kurbangaliev
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company