Re: Zedstore - compressed in-core columnar storage

From: Heikki Linnakangas <hlinnaka(at)iki(dot)fi>
To: Ashwin Agrawal <aagrawal(at)pivotal(dot)io>, Ashutosh Sharma <ashu(dot)coek88(at)gmail(dot)com>
Cc: DEV_OPS <devops(at)ww-it(dot)cn>, PostgreSQL mailing lists <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Zedstore - compressed in-core columnar storage
Date: 2019-08-15 10:05:49
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

We've continued hacking on Zedstore, here's a new patch version against
current PostgreSQL master (commit f1bf619acdf). If you want to follow
the development in real-time, we're working on this branch:

If you want to do performance testing with this, make sure you configure
with the --with-lz4 option. Otherwise, you'll get pglz compression,
which is *much* slower.

Major TODOs:

* Make it crash-safe, by WAL-logging.

* Teach the planner and executor to pass down the list of columns
needed. Currently, many plans will unnecessarily fetch columns that are
not needed.

* Make visibility checks against the TID tree in index-only scans.

* zedstore-toast pages are currently leaked, so you'll get a lot of
bloat if you delete/update rows with large datums

* Use the UNDO framework that's been discussed on another thread.
There's UNDO-logging built into zedstore at the moment, but it's not
very optimized.

* Improve free space management. Pages that become empty are currently
recycled, but space on pages that are not completely empty is not not
reused, and half-empty pages are not merged.

* Implement TID recycling. Currently, TIDs are allocated in increasing
order, and after all 2^48 TIDs have been used, even if the rows have
been deleted since, no more ruples can be inserted.

- Heikki

Attachment Content-Type Size
v3-zedstore.patch text/x-patch 1.5 MB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Konstantin Knizhnik 2019-08-15 11:01:26 Re: Built-in connection pooler
Previous Message Heikki Linnakangas 2019-08-15 09:38:30 Re: Zedstore - compressed in-core columnar storage