Re: [HACKERS] Custom compression methods

On Thu, Mar 4, 2021 at 4:03 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Thu, Mar 4, 2021 at 2:49 AM Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> >
> > Hi,
> >
> > Does this patch need to do something about ExtractReplicaIdentity()?
> > If there are compressed fields in the tuple being built, can we rely
> > on the decompression engine being available at the time we need to do
> > something with the tuple?
>
> We log the replica identity tuple in the WAL so that later walsender
> can stream this to the subscriber, and before sending to the
> subscriber anyway we have detoast all the data. Said that I think the
> problem you are worried about is not only with 'replica identity
> tuple' but it is with any tuple. I mean we copy the compressed field
> as it is in WAL and suppose we copy some fields which are compressed
> with lz4 and then we restart the server with another binary that is
> compiled without lz4. Now, the problem is the walsender can not
> decompress those

Based on the off list discussion with Robert, there are a couple of
problems which might be very difficult to handle when we support the
custom compression method using the access methods, the major problems
are 1) compressed data inside composite type 2) Access method might
get dropped before walsender decode the compressed data. I think the
first problem we still have is some solution although it may impact
performance in some cases i.e. extended record. But the problem of
the compressed data inside the WAL is a bigger problem. So as of now
we are planning to go ahead only with the built-in methods and if we
are only continuing with the built-in method so it doesn't make sense
to continue the access method infrastructure. I have rewrote the
patches without using the access method for compression. Changes in
the patches

- Removed complete dependency on the access method for compression
- While moving the tuple from one table to another table with
different compression method, no need to compare the compression
method and decompress.
- Alter table set compression, will not rewrite the old data, so only
the new tuple will be compressed with the new compression method.
- No preserve.

I feel the built-in method patch now looks cleaner and smaller than it
was before.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com