Re: Streaming base backups

On 05.01.2011 15:54, Magnus Hagander wrote:
> Attached is an updated streaming base backup patch, based off the work
> that Heikki started.
> ...
> I've implemented a frontend for this in pg_streamrecv, based on the assumption
> that we wanted to include this in bin/ for 9.1 - and that it seems like a
> reasonable place to put it. This can obviously be moved elsewhere if we want to.

Hmm, is there any point in keeping the two functionalities in the same
binary, taking the base backup and streaming WAL to an archive
directory? Looks like the only common option between the two modes is
passing the connection string, and the verbose flag. A separate
pg_basebackup binary would probably make more sense.

> That code needs a lot more cleanup, but I wanted to make sure I got the backend
> patch out for review quickly. You can find the current WIP branch for
> pg_streamrecv on my github page at https://github.com/mhagander/pg_streamrecv,
> in the branch "baserecv". I'll be posting that as a separate patch once it's
> been a bit more cleaned up (it does work now if you want to test it, though).

Looks like pg_streamrecv creates the pg_xlog and pg_tblspc directories,
because they're not included in the streamed tar. Wouldn't it be better
to include them in the tar as empty directories at the server-side?
Otherwise if you write the tar file to disk and untar it later, you have
to manually create them.

It would be nice to have an option in pg_streamrecv to specify the
backup label to use.

An option to stream the tar to stdout instead of a file would be very
handy too, so that you could pipe it directly to gzip for example. I
realize you get multiple tar files if tablespaces are used, but even if
you just throw an error in that case, it would be handy.

> * Suggestion from Heikki: perhaps at some point we're going to need a full
> bison grammar for walsender commands.

Maybe we should at least start using the lexer; we're not quite there to
need a full-blown grammar yet, but even a lexer might help.

BTW, looking at the WAL-streaming side of pg_streamrecv, if you start it
from scratch with an empty target directory, it needs to connect to
"postgres" database, to run pg_current_xlog_location(), and then
reconnect in replication mode. That's a bit awkward, there might not be
a "postgres" database, and even if there is, you might not have the
permission to connect to it. It would be much better to have a variant
of the START_REPLICATION command at the server-side that begins
streaming from the current location. Maybe just by leaving out the
start-location parameter.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com