Re: pg_streamrecv for 9.1?

On Wed, Dec 29, 2010 at 22:30, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>>> Would people be interested in putting pg_streamrecv
>>>> (http://github.com/mhagander/pg_streamrecv) in bin/ or contrib/ for
>>>> 9.1? I think it would make sense to do so.
>
> +1 for having that in core, only available for the roles WITH
> REPLICATION I suppose?

Yes.

Well, anybody who wants can run it, but they need those permissions on
the server to make it work. pg_streamrecv is entirely a client app.

>>> I think that the base backup feature is more important than simple streaming
>>> chunks of the WAL (SR already does this). Talking about the base backup over
>>> libpq, it is something we should implement to fulfill people's desire that
>>> claim an easy replication setup.
>>
>> Yes, definitely. But that also needs server side support.
>
> Yeah, but it's already in core for 9.1, we have pg_read_binary_file()
> there. We could propose a contrib module for previous version
> implementing the function in C, that should be pretty easy to code.

Oh. I didn't actually think about that one. So yeah, we could use that
- making it easy to code. However, I wonder how much less efficient it
would be than being able to stream the base backup. It's going to be a
*lot* more roundtrips across the network, and we're also going to
open/close the files a lot more.

Also, I haven't tested it, but a quick look at the code makes me
wonder how it will actually work with tablespaces - it seems to only
allow files under PGDATA? That could of course be changed..

>  The only reason I didn't do that for pg_basebackup is that I wanted a
>  self-contained python script, so that offering a public git repo is
>  all I needed as far as distributing the tool goes.

Right, there's an advantage with that when it comes to being able to
work on old versions.

>> Yeah, the WIP patch heikki posted is simliar, except it uses tar
>> format and is implemented natively in the backend with no need for
>> pl/pythonu to be installed.
>
> As of HEAD the dependency on pl/whatever is easily removed.
>
> The included C tool would need to have a parallel option from the get-go
> if at all possible, but if you look at the pg_basebackup prototype, it
> would be good to drop the wrong pg_xlog support in there and rely on a
> proper archiving setup on the master.
>
> Do you want to work on internal archive and restore commands over libpq
> in the same effort too?  I think this tool should be either a one time
> client or a daemon with support for:

Definitely a one-time client. If you want it to be a deamon, you write
a small wrapper that makes it one :)

>  - running a base backup when receiving a signal
>  - continuous WAL streaming from a master

Yes.

>  - accepting standby connections and streaming to them

I see that as a separate tool, I think. But still a useful one, sure.

>  - one-time libpq "streaming" of a WAL file, either way

Hmm. That might be interesting, yes.

> Maybe we don't need to daemonize the tool from the get-go, but if you're
> going parallel for the base-backup case you're almost there, aren't you?
> Also having internal commands for archive and restore commands that rely
> on this daemon running would be great too.

I don't want anything *relying* on this tool. I want to keep the
current way where you can choose whatever you prefer - I just want us
to ship a good default tool.

> I'd offer more help if it wasn't for finishing the extension patches,

:-) Yeah, focus on that, please - don't want to get it stalled.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/