Re: Allowing multiple concurrent base backups

On Wed, Jan 12, 2011 at 10:26:05AM +0100, marcin mank wrote:
> On Tue, Jan 11, 2011 at 8:07 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > Magnus Hagander <magnus(at)hagander(dot)net> writes:
> >> On Tue, Jan 11, 2011 at 19:51, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >>> Seems like either one of these is fairly problematic in that you have to
> >>> have some monstrous kluge to get the backup_label file to appear with
> >>> the right name in the tarfile.  How badly do we actually need this?
> >>> I don't think the use-case for concurrent base backups is all that large
> >>> in practice given the I/O hit it's going to involve.
> >
> >> I think it can be done cleaner in the tar file injection - I've been
> >> chatting with Heikki offlist about that. Not sure, but I have a
> >> feeling it does.
> >
> > One point that I'm particularly interested to see how you'll kluge it
> > is ensuring that the tarball contains only the desired temp data and not
> > also the "real" $PGDATA/backup_label, should there be a normal base
> > backup being done concurrently with the streamed one.
> >
> > The whole thing just seems too fragile and dangerous to be worth dealing
> > with given that actual usage will be a corner case.  *I* sure wouldn't
> > trust it to work when the chips were down.
>
> Maybe if pg_start_backup() notices that there is another backup
> running should block waiting for another session to run
> pg_stop_backup() ? Or have a new function like pg_start_backup_wait()
> ?
>
> Considering that parallell base backups would be io-bound (or
> network-bound), there is little need to actually run them in parallell

That's not actually true. Backups at the moment are CPU-bound, and
running them in parallel is one way to make them closer to I/O-bound,
which is what they *should* be.

There are other proposals out there, and some work being done, to make
backups less dependent on CPU, among them:

- Making the on-disk representation smaller
- Making COPY more efficient

As far as I know, none of this work is public yet.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate