Re: [HACKERS] Proposal: generic WAL compression

On 07/02/18 19:42, Stephen Frost wrote:
>> really useful for my "OUZO" project (a fork of the RUM access method).
>
> Glad to hear that you're continuing to work on it.

Yes, it will be available on Github eventually.

>> One general comment I can already make is that enabling compression
>> should be made optional, which appears to be a small and easy addition
>> to the generic WAL API.
>
> The question would be- is there a way for us to determine when it should
> be enabled or not enabled, or do we have to ask the user to tell us?

My suggestion would be adding something like

#define GENERIC_XLOG_MOVE_DETECT_ON 0x0002 /* search for moved
parts of page image */
#define GENERIC_XLOG_MOVE_DETECT_OFF 0x0004 /* do not search for
moved parts of page image */

This would be used for the 'flags' argument of GenericXLogRegisterBuffer(),
allowing code using generic WAL (such as custom access methods) to
control this kind of compression on a per buffer basis, if need be.
Then it would be up to the author of any extension relying on generic
WAL to decide if and how the user will be given control of WAL
compression.

However,

> Asking the user is something we'd really not do unless we absolutely
> have to. Much better is if we can figure out when it's best to enable
> or disable (or use/not use) based on some criteria and do so
> automatically.

this is a point I did not think of before :-) Probably a logical
choice would be having "GENERIC_XLOG_MOVE_DETECT" default to true
for a 'wal_level' value of 'replica' (the default setting nowadays)
or higher, and default to false for 'minimal'.
In this way, generic WAL will be automatically compressed when it is
potentially stored for some longer time, or consuming network bandwidth.

--
Markus Nullmeier http://www.g-vo.org
German Astrophysical Virtual Observatory (GAVO)