Re: Quick-and-dirty compression for WAL backup blocks

Junji TERAMOTO <teramoto(dot)junji(at)lab(dot)ntt(dot)co(dot)jp> writes:
> In the XLogInsert(), it makes two kinds of logs, "whole buffer(page)
> log" and "partial buffer log", isn't it? Is it only "who buffer log"
> to generate a log with "hole"?

Right.

> Tom Lane wrote:
>> The overhead needed is only 2 bytes to show the number of
>> bytes removed.

> In "whole buffer log", there is a page header that includes offset of
> "hole" (lower and upper). If we use that information, we don't need
> any overhead, do we?

No, because the WAL code cannot assume that all pages follow the
convention that pd_lower and pd_upper represent the boundaries of
free space. (As a counterexample: index metapages don't always
do that.) I think the transformation has to be guaranteed lossless,
which means that at a minimum you'd need to check whether the data
in between pd_lower and pd_upper really is zeroes. So the irreducible
minimum overhead is 1 bit to tell whether you compressed or not.
Considering alignment requirements, you might as well expend a couple
of bytes and keep it simple. (In the patch as committed, I ended up
using 4 bytes --- a uint16 hole start and a uint16 hole length ---
because it kept the code simple. The alignment requirements mean the
extra 2 bytes are usually free anyway.)

regards, tom lane