Re: [HACKERS][Proposal] LZ4 Compressed Storage Manager

On Sun, Mar 31, 2019 at 05:25:51PM +0300, Николай Петров wrote:
> Why it should be implemented on Storage Manager level instead of usage
> Pluggable storage API [9]?
> - From my perspective view Storage Manager level implementation
> allows to focus on proper I/O operations and compression.
> It allows to write much more simple realization. It's because of
> Pluggable storage API force you to implement more complex
> interfaces. To be honest, I am really hesitating about this point,
> especially because of Pluggable storage API allows to create
> extension without core code modification and it potentially allows
> to use more perfective compression algorithms (Table Access Manager
> allows you to get more information about storing data).
>
> I would like to implement a proof of concept
> and have a couple of questions:
> - your opinion about necessity of this feature
> (Compressed Storage Manager)
> - Is it good idea to implement DB compression on Storage Manager
> level? Perhaps it is better to use Pluggable storage API.
> - Is there any reason to refuse this proposal?
> - Are there any circumstances what didn't allow to implement
> Compressed Storage Manager?

Stepping back a bit, there are several levels of compression:

1. single field
2. across all fields in a row
3. across rows on a single page
4. across all rows in a table
5. across tables in a database

We currently do #1 with TOAST, and your approach would allow the first
three. #4 feels like it is getting near the features of columnar
storage. I think it is unclear if adding #2 and #3 produce enough of a
benefit to warrant special storage, given the complexity and overhead of
implementing it.

I do think the Pluggable storage API is the right approach, and, if you
are going to go that route, adding #4 compression seems very worthwhile.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +