Re: [HACKERS] Custom compression methods

From: Dilip Kumar <dilipbalaut(at)gmail(dot)com>
To: Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, Alexander Korotkov <a(dot)korotkov(at)postgrespro(dot)ru>, David Steele <david(at)pgmasters(dot)net>, Ildus Kurbangaliev <i(dot)kurbangaliev(at)gmail(dot)com>, Dmitry Dolgov <9erthalion6(at)gmail(dot)com>, PostgreSQL Developers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: [HACKERS] Custom compression methods
Date: 2020-11-11 14:39:37
Message-ID: CAFiTN-vwVHdGi36qoF_ZRW5ehD+ea642XH=+EBk50Qc4V1h1TA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Sun, Nov 8, 2020 at 4:29 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Thu, Oct 29, 2020 at 12:07 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> >
> > On Thu, Oct 29, 2020 at 12:31 AM Tomas Vondra
> > <tomas(dot)vondra(at)2ndquadrant(dot)com> wrote:
> > >
> > > On Wed, Oct 28, 2020 at 01:16:31PM +0530, Dilip Kumar wrote:
> > > >>
> > > >> ...
> > > >>
> > > >> I have added the next patch to support the compression options. I am
> > > >> storing the compression options only for the latest compression
> > > >> method. Basically, based on this design we would be able to support
> > > >> the options which are used only for compressions. As of now, the
> > > >> compression option infrastructure is added and the compression options
> > > >> for inbuilt method pglz and the external method zlib are added. Next,
> > > >> I will work on adding the options for the lz4 method.
> > > >
> > > >In the attached patch set I have also included the compression option
> > > >support for lz4. As of now, I have only supported the acceleration
> > > >for LZ4_compress_fast. There is also support for the dictionary-based
> > > >compression but if we try to support that then we will need the
> > > >dictionary for decompression also. Since we are only keeping the
> > > >options for the current compression methods, we can not support
> > > >dictionary-based options as of now.
> > > >
> > >
> > > OK, thanks. Do you have any other plans to improve this patch series? I
> > > plan to do some testing and review, but if you're likely to post another
> > > version soon then I'd wait a bit.
> >
> > There was some issue in create_compression_method.sgml and the
> > drop_compression_method.sgml was missing. I have fixed that in the
> > attached patch. Now I am not planning to change anything soon so you
> > can review. Thanks.
>
> The patches were not applying on the current head so I have re-based them.

There were a few problems in this rebased version, basically, the
compression options were not passed while compressing values from the
brin_form_tuple, so I have fixed this.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Attachment Content-Type Size
v14-0005-new-compression-method-extension-for-zlib.patch application/octet-stream 9.8 KB
v14-0002-alter-table-set-compression.patch application/octet-stream 12.9 KB
v14-0001-Built-in-compression-method.patch application/octet-stream 200.9 KB
v14-0003-Add-support-for-PRESERVE.patch application/octet-stream 40.6 KB
v14-0004-Create-custom-compression-methods.patch application/octet-stream 44.8 KB
v14-0006-Support-compression-methods-options.patch application/octet-stream 49.7 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Stephen Frost 2020-11-11 14:56:44 Re: In-place persistance change of a relation
Previous Message Stephen Frost 2020-11-11 14:36:24 Re: Disable WAL logging to speed up data loading