Re: Compress prune/freeze records with Delta Frame of Reference algorithm

Hello hackers,

> Can this DFoR code replace integerset.c easily? Can we use it for
> the vacuum dead TID list? For GIN posting lists? Where else?

Heikki, thank you for your attention and proposals. I'm learning areas
you proposed to be developed. This took time, since I am not adept at
them. Last week I also have been developing the DFoR patch to support
unsorted sequences. That's why there was the delay in answering.

About GIN.
Since GIN exploits TIDs sequences and saves it on the disk, it can be
the most appropriate candidate to be developed with DFoR.

About the dead TID list.
If I'm not mistaken, the dead TID list exists only in RAM and never on
the disk or in the network. So, what is the advantage supposed to be
achieved due to using compression in the dead TID list?

About the GiST vacuuming and the use of integerset in it.
The integerset implements a tree in addition to compression.
DFoR now performs only compression. Moreover the size of a pack is
flexible (varying), which must become an issue for its usage in the
tree. It needs more thorough further elaboration to be developed.

So what do you think about improving GIN by means of DFOR? Should I try?

Best regards,
Evgeny Voropaev,
Tantor Labs, LLC.

P.S.
In the v12 version of the patch:
- implemented the DFOR compression for unsorted sequences;
- implemented the compression of frozen and redirected tuple offsets
in the prune/freeze WAL record
- ignored header checking of header templates from DFOR file set;
- rebased onto 9b43e6793b0.