Re: Zedstore - compressed in-core columnar storage

On Mon, Apr 15, 2019 at 10:50:21AM -0700, Ashwin Agrawal wrote:
> On Mon, Apr 15, 2019 at 10:33 AM Tomas Vondra
> <tomas(dot)vondra(at)2ndquadrant(dot)com> wrote:
>
> ...
>
> I see. Perhaps it'd be better to call the flag ZSBT_CONTAINER, when it
> means "this is a container". And then have another flag to track whether
> the container is compressed or not. But as I suggested elsewhere in this
> thread, I think it might be better to store some ID of the compression
> algorithm used instead of a simple flag.
>
> FWIW when I had to deal with incremental compression (adding data into
> already compressed buffers), which is what seems to be happening here, I
> found it very useful/efficient to allow partially compressed buffers and
> only trigger recompressin when absolutely needed.
>
> Applied to this case, the container would first store compressed chunk,
> followed by raw (uncompressed) data. Say, like this:
>
> ZSContainerData {
>
>     // header etc.
>
>     int nbytes;         /* total bytes in data */
>     int ncompressed;    /* ncompressed <= nbytes, fully compressed when
>                          * (ncompressed == nbytes) */
>
>     char data[FLEXIBLE_ARRAY_MEMBER];
> }
>
> When adding a value to the buffer, it'd be simply appended to the data
> array. When the container would grow too much (can't fit on the page or
> something), recompression is triggered.
>
> I think what you suggested here is exactly how its handled currently, just
> the mechanics are little different. Plain items are added to page as
> insertions are performed. Then when page becomes full, compression is
> triggerred container item is created for them to store the compressed
> data. Then new insertions are stored as plain items, once again when page
> becomes full, they are compressed and container item created for it. So,
> never, compressed data is attempted to be compressed again. So, on page
> plain items are acting as data section you mentioned above. A page can
> have mix of n plain and n container items.

Maybe. I'm not going to pretend I fully understand the internals. Does
that mean the container contains ZSUncompressedBtreeItem as elements? Or
just the plain Datum values?

regards

--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services