Skip site navigation (1) Skip section navigation (2)

Re: Faster compression, again

From: Daniel Farina <daniel(at)heroku(dot)com>
To: Simon Riggs <simon(at)2ndquadrant(dot)com>
Cc: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Faster compression, again
Date: 2012-03-15 22:34:35
Message-ID: CAAZKuFbaH2N+bSysom7gsgq+S_nffgGNb0do1KLBSHo05_QKkQ@mail.gmail.com (view raw or flat)
Thread:
Lists: pgsql-hackers
On Thu, Mar 15, 2012 at 3:14 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Wed, Mar 14, 2012 at 6:06 PM, Daniel Farina <daniel(at)heroku(dot)com> wrote:
>
>> If we're curious how it affects replication
>> traffic, I could probably gather statistics on LZO-compressed WAL
>> traffic, of which we have a pretty huge amount captured.
>
> What's the compression like for shorter chunks of data? Is it worth
> considering using this for the libpq copy protocol and therefore
> streaming replication also?

The overhead is between 1 and 5 bytes that reserve the high bit as a
continuation bit (so one byte for small data), and then straight into
data.  So I think it could be applied for most payloads that are a few
bytes wide.  Presumably that could be lifted, but the format
description only allows for 2**32 - 1 for the uncompressed size.

I'd really like to find a way to layer both message-oblivious and
message-aware transport under FEBE with both backend and frontend
support without committing the project to new code for-ever-and-ever.
I guess I could investigate it in brief now, unless you've already
thought about/done some work in that area.

-- 
fdr

In response to

Responses

pgsql-hackers by date

Next:From: ktm@rice.eduDate: 2012-03-15 22:40:09
Subject: Re: Faster compression, again
Previous:From: Simon RiggsDate: 2012-03-15 22:23:32
Subject: Re: foreign key locks, 2nd attempt

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group