Re: Streaming base backups

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Attached is an updated streaming base backup patch, based off the work

Thanks! :)

> * Compression: Do we want to be able to compress the backups server-side? Or
> defer that to whenever we get compression in libpq? (you can still tunnel it
> through for example SSH to get compression if you want to) My thinking is
> defer it.

Compression in libpq would be a nice way to solve it, later.

> * Compression: We could still implement compression of the tar files in
> pg_streamrecv (probably easier, possibly more useful?)

What about pg_streamrecv | gzip > …, which has the big advantage of
being friendly to *any* compression command line tool, whatever patents
and licenses?

> * Stefan mentiond it might be useful to put some
> posix_fadvise(POSIX_FADV_DONTNEED)
> in the process that streams all the files out. Seems useful, as long as that
> doesn't kick them out of the cache *completely*, for other backends as well.
> Do we know if that is the case?

Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
not already in SHM?

> * include all the necessary WAL files in the backup. This way we could generate
> a tar file that would work on it's own - right now, you still need to set up
> log archiving (or use streaming repl) to get the remaining logfiles from the
> master. This is fine for replication setups, but not for backups.
> This would also require us to block recycling of WAL files during the backup,
> of course.

Well, I would guess that if you're streaming the WAL files in parallel
while the base backup is taken, then you're able to have it all without
archiving setup, and the server could still recycling them.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support