Skip site navigation (1) Skip section navigation (2)

Compression of full-page-writes

From: Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
To: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Compression of full-page-writes
Date: 2013-08-30 02:55:54
Message-ID: CAHGQGwGqG8e9YN0fNCUZqTTT=hNr7Ly516kfT5ffqf4pp1qnHg@mail.gmail.com (view raw or flat)
Thread:
Lists: pgsql-hackers
Hi,

Attached patch adds new GUC parameter 'compress_backup_block'.
When this parameter is enabled, the server just compresses FPW
(full-page-writes) in WAL by using pglz_compress() before inserting it
to the WAL buffers. Then, the compressed FPW is decompressed
in recovery. This is very simple patch.

The purpose of this patch is the reduction of WAL size.
Under heavy write load, the server needs to write a large amount of
WAL and this is likely to be a bottleneck. What's the worse is,
in replication, a large amount of WAL would have harmful effect on
not only WAL writing in the master, but also WAL streaming and
WAL writing in the standby. Also we would need to spend more
money on the storage to store such a large data.
I'd like to alleviate such harmful situations by reducing WAL size.

My idea is very simple, just compress FPW because FPW is
a big part of WAL. I used pglz_compress() as a compression method,
but you might think that other method is better. We can add
something like FPW-compression-hook for that later. The patch
adds new GUC parameter, but I'm thinking to merge it to full_page_writes
parameter to avoid increasing the number of GUC. That is,
I'm thinking to change full_page_writes so that it can accept new value
'compress'.

I measured how much WAL this patch can reduce, by using pgbench.

* Server spec
  CPU: 8core, Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
  Mem: 16GB
  Disk: 500GB SSD Samsung 840

* Benchmark
  pgbench -c 32 -j 4 -T 900 -M prepared
  scaling factor: 100

  checkpoint_segments = 1024
  checkpoint_timeout = 5min
  (every checkpoint during benchmark were triggered by checkpoint_timeout)

* Result
  [tps]
  1386.8 (compress_backup_block = off)
  1627.7 (compress_backup_block = on)

  [the amount of WAL generated during running pgbench]
  4302 MB (compress_backup_block = off)
  1521 MB (compress_backup_block = on)

At least in my test, the patch could reduce the WAL size to one-third!

The patch is WIP yet. But I'd like to hear the opinions about this idea
before completing it, and then add the patch to next CF if okay.

Regards,

-- 
Fujii Masao

Attachment: compress_fpw_v1.patch
Description: application/octet-stream (4.4 KB)

Responses

pgsql-hackers by date

Next:From: Satoshi NagayasuDate: 2013-08-30 03:07:45
Subject: Re: Compression of full-page-writes
Previous:From: wangshuoDate: 2013-08-30 01:57:54
Subject: ENABLE/DISABLE CONSTRAINT NAME

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group