Re: [FEATURE REQUEST] Streaming Onlinebackup (Maybe OFFTOPIC)

On Thu, 2007-09-06 at 19:23 -0400, Tom Lane wrote:
> Jeff Davis <pgsql(at)j-davis(dot)com> writes:
> > On Thu, 2007-09-06 at 12:08 +0100, Heikki Linnakangas wrote:
> >> With file-based log shipping, you can get down to 1 second, by using the
> >> archive_timeout setting. It will produce a lot of log files with very
> >> little content in them, but they will compress well.
>
> > I tried doing a couple pg_switch_xlog() in quick succession, and the WAL
> > files that represent less than a second of actual data don't seem much
> > more compressible than the rest.
>
> That's because we do not try to zero out a WAL file when recycling it,
> so the part after what's been used is not any more compressible than the
> valid part.
>

That was my point.

> pg_switch_xlog is not, and was never intended to be, a solution for a
> low-latency log-shipping scheme. The right solution for that is to make
> a smarter log-shipping daemon that transmits only part of a WAL segment
> at a time. (As Hannu is getting tired of repeating, you can find a
> working example in SkyTools.)

I think that in addition to WalMgr (which is the tool I assume you're
referring to), there's still room for a simple tool that can zero out
the meaningless data in a partially-used WAL segment before compression.
It seems reasonable to me, so long as you keep archive_timeout at
something reasonably high.

If nothing else, people that already have a collection of archived WAL
segments would then be able to compact them.

I agree that something like WalMgr is better for low-latency, however.

Regards,
Jeff Davis