Compressing temporary files

From: Andrey Borodin <x4mmm(at)yandex-team(dot)ru>
To: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Cc: "Chainani, Naresh" <nareshkc(at)amazon(dot)com>
Subject: Compressing temporary files
Date: 2021-09-11 12:31:37
Message-ID: 4F368CFD-306C-483E-968E-072D225A8DCE@yandex-team.ru
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi hackers!

There's a lot of compression discussions nowadays. And that's cool!
Recently Naresh Chainani in private discussion shared with me the idea to compress temporary files on disk.
And I was thrilled to find no evidence of implementation of this interesting idea.

I've prototyped Random Access Compressed File for fun[0]. The code is very dirty proof-of-concept.
I compress Buffile by one block at a time. There are directory pages to store information about the size of each compressed block. If any byte of the block is changed - whole block is recompressed. Wasted space is never reused. If compressed block is more then BLCSZ - unknown bad things will happen :)

Here are some my observations.

0. The idea seems feasible. API of fd.c used by buffile.c can easily be abstracted for compressed temporary files. Seeks are necessary, but they are not very frequent. It's easy to make temp file compression GUC-controlled.

1. Temp file footprint can be easily reduced. For example query
create unlogged table y as select random()::text t from generate_series(0,9999999) g;
uses for toast index build 140000000 bytes of temp file. With patch this value is reduced to 40841704 (x3.42 smaller).

2. I have not found any evidence of performance improvement. I've only benchmarked patch on my laptop. And RAM (page cache) diminished any difference between writing compressed block and uncompressed block.

How do you think: does it worth to pursue the idea? OLTP systems rarely rely on data spilled to disk.
Are there any known good random access compressed file libs? So we could avoid reinventing the wheel.
Maybe someone tried this approach before?

Thanks!

Best regards, Andrey Borodin.

[0] https://github.com/x4m/postgres_g/commit/426cd767694b88e64f5e6bee99fc653c45eb5abd

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tom Lane 2021-09-11 16:19:49 Re: missing warning in pg_import_system_collations
Previous Message Zhihong Yu 2021-09-11 08:28:42 Re: Polyphase merge is obsolete