Re: Zedstore - compressed in-core columnar storage

On Thu, Aug 15, 2019 at 01:05:49PM +0300, Heikki Linnakangas wrote:
> We've continued hacking on Zedstore, here's a new patch version against
> current PostgreSQL master (commit f1bf619acdf). If you want to follow the
> development in real-time, we're working on this branch:
> https://github.com/greenplum-db/postgres/tree/zedstore

Thanks for persuing this. It's an exciting development and I started
looking at how we'd put it to use. I imagine we'd use it in favour of ZFS
tablespaces, which I hope to retire.

I've just done very brief experiment so far. Some thoughts:

. I was missing a way to check for compression ratio; it looks like zedstore
with lz4 gets ~4.6x for our largest customer's largest table. zfs using
compress=gzip-1 gives 6x compression across all their partitioned tables,
and I'm surprised it beats zedstore .

. What do you think about pg_restore --no-tableam; similar to
--no-tablespaces, it would allow restoring a table to a different AM:
PGOPTIONS='-c default_table_access_method=zedstore' pg_restore --no-tableam ./pg_dump.dat -d postgres
Otherwise, the dump says "SET default_table_access_method=heap", which
overrides any value from PGOPTIONS and precludes restoring to new AM.

. It occured to me that indices won't be compressed. That's no fault of
zedstore, but it may mean that some sites would need to retain their ZFS
tablespace, and suggests the possibility of an separate, future project
(I wonder if there's some way a new meta-AM could "enable" compression of
other index AMs, to avoid the need to implement zbtree, zhash, zgin, ...).

. it'd be nice if there was an ALTER TABLE SET ACCESS METHOD, to allow
migrating data. Otherwise I think the alternative is:
begin; lock t;
CREATE TABLE new_t LIKE (t INCLUDING ALL) USING (zedstore);
INSERT INTO new_t SELECT * FROM t;
for index; do CREATE INDEX...; done
DROP t; RENAME new_t (and all its indices). attach/inherit, etc.
commit;

. Speaking of which, I think LIKE needs a new option for ACCESS METHOD, which
is otherwise lost.

Cheers,
Justin