Re: Making pg_standby compression-friendly

On Thu, Oct 23, 2008 at 1:15 AM, Heikki Linnakangas <
heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:

> Charles Duffy wrote:
>
>> I'm interested in compressing archived WAL segments in an environment
>> set up for PITR in the interests of reducing both network traffic and
>> storage requirements. However, pg_standby presently checks file sizes,
>> requiring that an archive segment be exactly the right size to be
>> considered valid. The idea of compressing log segments is not new --
>> the clearxlogtail project in pgfoundry provides a tool to make such
>> compression more effective, and is explicitly intended for said
>> purpose -- but as of 8.3.4, pg_standby appears not to support such
>> environments; I propose adding such support.
>>
>
> Can't you decompress the files in whatever script you use to copy them to
> the archive location?

To be sure I understand -- you're proposing a scenario in which the
archive_command on the master compresses the files, passes them over to the
slave while compressed, and then decompresses them on the slave for storage
in their decompressed state? That succeeds in the goal of decreasing network
bandwidth, but (1) isn't necessarily easy to implement over NFS, and (2)
doesn't succeed in decreasing storage requirements on the slave.

(While pg_standby's behavior is to delete segments which are no longer
needed to keep a warm standby slave running, I maintain a separate archive
for PITR use with hardlinked copies of those same archive segments; storage
on the slave is a much bigger issue in this environment than it would be if
the space used for segments were being deallocated as soon as pg_standby
chose to unlink them).

[Heikki, please accept my apologies for the initial off-list response; I
wasn't paying enough attention to gmail's default reply behavior].