Skip site navigation (1) Skip section navigation (2)

Faster compression, again

From: Daniel Farina <daniel(at)heroku(dot)com>
To: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Faster compression, again
Date: 2012-03-14 18:06:16
Message-ID: CAAZKuFb59sABSa7gCG0vnVnGb-mJCUBBbrKiyPraNXHnis7KMw@mail.gmail.com (view raw or flat)
Thread:
Lists: pgsql-hackers
For 9.3 at a minimum.

The topic of LZO became mired in doubts about:

* Potential Patents
* The author's intention for the implementation to be GPL

Since then, Google released "Snappy," also an LZ77-class
implementation, and it has been ported to C (recently, and with some
quirks, like no LICENSE file...yet, although it is linked from the
original Snappy project).  The original Snappy (C++) has a BSD license
and a patent grant (which shields you from Google, at least).  Do we
want to investigate a very-fast compression algorithm inclusion again
in the 9.3 cycle?

I've been using the similar implementation "LZO" for WAL archiving and
it is a significant savings (not as much as pg_lesslog, but also less
invasive).  It is also fast enough that even if one were not to uproot
TOAST's compression that it would probably be very close to a complete
win for protocol traffic, whereas SSL's standardized zlib can
definitely be a drag in some cases.

This idea resurfaces often, but the reason why I wrote in about it is
because I have a table which I categorized as "small" but was, in
fact, 1.5MB, which made transferring it somewhat slow over a remote
link.  zlib compression takes it down to about 550K and lzo (similar,
but not identical) 880K.  If we're curious how it affects replication
traffic, I could probably gather statistics on LZO-compressed WAL
traffic, of which we have a pretty huge amount captured.

-- 
fdr

Responses

pgsql-hackers by date

Next:From: ktm@rice.eduDate: 2012-03-14 18:13:27
Subject: Re: Faster compression, again
Previous:From: Marco NenciariniDate: 2012-03-14 18:03:08
Subject: Re: [PATCH] Support for foreign keys with arrays

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group