Re: Improving compressibility of WAL files

On Fri, 9 Jan 2009, Aidan Van Dyk wrote:

> *I* didn't see an easy way to get at the "written" size later on in the
> chain (i.e. in the actual archiving), so I took the path of least
> resitance.

I was hoping it might fall out of the other work being done in that area,
given how much that code is still being poked at right now. As Hannu
pointed out, from a conceptual level you just need to carry along the same
information that pg_current_xlog_location() returns to the archiver on all
the paths where a segment might end early.

> If I wrapped this zeroing in GUC, people could choose wether to pay the
> penalty or not, would that satisfy anyone?

Rather than creating a whole new GUC, it might it be possible to turn
archive_mode into an enum setting: off, on, and cleaned as the modes
perhaps. That would avoid making a new setting, with the downside that a
bunch of critical code would look less clear than it does with a boolean.

> Again, *I* think that the force_switch case is going to happen when the
> admin's quite happy to pay that penalty... But obviously not
> everyone...

I understand the case you've made for why it doesn't matter, and for
almost every case you're right. The situation it may be vulnerable to is
where a burst of transactions come in just as the archive timeout expires
after minimal WAL activity. There I think you can end up with a bunch of
clients waiting behind an almost full zero fill operation, which pushes up
the worst-case latency. I've been able to measure the impact of the
similar case where zero-filling a brand new segment can impact things;
this would be much less like to happen because the timing would have to
line up just wrong, but I think it's still possible.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD