Nicos Panayides <nicos(at)magneta(dot)com(dot)cy> wrote:
> The database generates about 3 PITR log files per minute. If my
> calculations are correct the sites need to be connected with a
> 7MBit connection and the logs will need about 68GB of storage per
> day!
I'd have put the minimum line speed at 8Mb/second, but I figure 10
bits per byte to allow for parity and packet overhead, and the
assumption that I might want to use the line for something else
(like monitoring) at the same time; but yeah, those numbers are in
the right ballpark.
> Does anyone have any suggestions on how to significantly reduce
> the volume of log files or recommend another off-site backup
> solution that would require less bandwidth and storage?
We stream the WAL files through gzip in our archive script. That
reduces them to 4MB to 8MB during normal usage, and 12MB to 15MB
during our routine database vacuums. If you have relatively idle
periods during which the log is pushed by archive_timeout, you
should filter through pg_clearxlogtail or pglesslog before using
gzip; the former yields a 16kB file during idle periods. (I'm not
sure about the other, but it should be capable of getting even
smaller.)
You'll need to determine how far back it's valuable to have the
"point in time recovery" capabilities. We do weekly base backups
and keep WAL files to restore forward from the earlier to current,
but then keep monthly "archival" backups, where each base backup is
kept with just the WAL files needed to successfully restore it.
-Kevin