Re: Proposal: Incremental Backup

On Thu, Aug 7, 2014 at 11:03:40AM +0100, Simon Riggs wrote:
> Well, there is a huge difference between file-level and block-level backup.
>
> Designing, writing and verifying block-level backup to the point that
> it is acceptable is a huge effort. (Plus, I don't think accumulating
> block numbers as they are written will be "low overhead". Perhaps
> there was a misunderstanding there and what is being suggested is to
> accumulate file names that change as they are written, since we
> already do that in the checkpointer process, which would be an option
> between 2 and 3 on the above list).
>
> What is being proposed here is file-level incremental backup that
> works in a general way for various backup management tools. It's the
> 80/20 first step on the road. We get most of the benefit, it can be
> delivered in this release as robust, verifiable code. Plus, that is
> all we have budget for, a fairly critical consideration.
>
> Big features need to be designed incrementally across multiple
> releases, delivering incremental benefit (or at least that is what I
> have learned). Yes, working block-level backup would be wonderful, but
> if we hold out for that as the first step then we'll get nothing
> anytime soon.

That is fine. I just wanted to point out that as features are added,
file-level incremental backups might not be useful. In fact, I think
there are a lot of users for which file-level incremental backups will
never be useful, i.e. you have to have a lot of frozen/static data for
file-level incremental backups to be useful.

I am a little worried that many users will not realize this until they
try it and are disappointed, e.g. "Why is PG writing to my static data
so often?" --- then we get beaten up about our hint bits and freezing
behavior. :-(

I am just trying to set realistic expectations.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +