|From:||Konstantin Knizhnik <k(dot)knizhnik(at)postgrespro(dot)ru>|
|To:||Ashwin Agrawal <aagrawal(at)pivotal(dot)io>, Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz>|
|Cc:||PostgreSQL mailing lists <pgsql-hackers(at)postgresql(dot)org>|
|Subject:||Re: Zedstore - compressed in-core columnar storage|
|Views:||Raw Message | Whole Thread | Download mbox | Resend email|
On 11.04.2019 8:03, Ashwin Agrawal wrote:
>> On Apr 10, 2019, at 9:08 PM, Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> wrote:
>>> On 11/04/19 4:01 PM, Mark Kirkwood wrote:
>>>> On 9/04/19 12:27 PM, Ashwin Agrawal wrote:
>>>> Heikki and I have been hacking recently for few weeks to implement
>>>> in-core columnar storage for PostgreSQL. Here's the design and initial
>>>> implementation of Zedstore, compressed in-core columnar storage (table
>>>> access method). Attaching the patch and link to github branch  to
>>>> follow along.
>>> Very nice. I realize that it is very early days, but applying this patch I've managed to stumble over some compression bugs doing some COPY's:
>>> benchz=# COPY dim1 FROM '/data0/dump/dim1.dat'
>>> USING DELIMITERS ',';
>>> psql: ERROR: compression failed. what now?
>>> CONTEXT: COPY dim1, line 458
>>> The log has:
>>> 2019-04-11 15:48:43.976 NZST  ERROR: XX000: compression failed. what now?
>>> 2019-04-11 15:48:43.976 NZST  CONTEXT: COPY dim1, line 458
>>> 2019-04-11 15:48:43.976 NZST  LOCATION: zs_compress_finish, zedstore_compression.c:287
>>> 2019-04-11 15:48:43.976 NZST  STATEMENT: COPY dim1 FROM '/data0/dump/dim1.dat'
>>> USING DELIMITERS ',';
>>> The dataset is generated from and old DW benchmark I wrote (https://urldefense.proofpoint.com/v2/url?u=https-3A__sourceforge.net_projects_benchw_&d=DwIDaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=gxIaqms7ncm0pvqXLI_xjkgwSStxAET2rnZQpzba2KM&m=BgmTkDoY6SKOgODe8v6fpH4hs-wM0H91cLfrAfEL6C0&s=lLcXp_8h2bRb_OR4FT8kxD-FG9MaLBPU7M5aV9nQ7JY&e=). The row concerned looks like:
>>> 457,457th interesting measure,1th measure type,aqwycdevcmybxcnpwqgrdsmfelaxfpbhfxghamfezdiwfvneltvqlivstwralshsppcpchvdkdbraoxnkvexdbpyzgamajfp
>>> 458,458th interesting measure,2th measure type,bjgdsciehjvkxvxjqbhtdwtcftpfewxfhfkzjsdrdabbvymlctghsblxucezydghjrgsjjjnmmqhncvpwbwodhnzmtakxhsg
>>> I'll see if changing to LZ4 makes any different.
>> The COPY works with LZ4 configured.
> Thank you for trying it out. Yes, noticed for certain patterns pg_lzcompress() actually requires much larger output buffers. Like for one 86 len source it required 2296 len output buffer. Current zedstore code doesn’t handle this case and errors out. LZ4 for same patterns works fine, would highly recommend using LZ4 only, as anyways speed is very fast as well with it.
Internal Postgres lz compressor is really very inefficient comparing
with other compression algorithms.
But in any case you should never assume that size of compressed data
will be smaller than size of plain data.
Moreover, if you are trying to compress already compressed data, then
result almost always will be larger.
If size of compressed data is larger (or even not significantly smaller)
than size of raw data, then you should store original data.
lz4 is actually very fast. But it doesn't provide good compression ratio.
This my results of compressing pbench data using different compressors:
Configuration Size (Gb) Time (sec)
zlib (default level) 2.37 284
zlib (best speed) 2.43 191
postgres internal lz 3.89 214
snappy 5.18 99
lzfse 2.80 1099
(apple) 2.80 1099
You see that zstd provides almost 2 times better compression ration and
almost at the same speed.
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
|Next Message||Higuchi, Daisuke||2019-04-11 09:03:42||RE: Problem during Windows service start|
|Previous Message||Magnus Hagander||2019-04-11 08:33:13||Re: pg_rewind vs superuser|