Re: Streaming base backups

Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> Compression in libpq would be a nice way to solve it, later.
>
> Yeah, I'm pretty much set on postponing that one.

+1, in case it was not clear for whoever's counting the votes :)

>> What about pg_streamrecv | gzip > …, which has the big advantage of
>
> That's part of what I meant with "easier and more useful".

Well…

> Right now though, pg_streamrecv will output one tar file for each
> tablespace, so you can't get it on stdout. But that can be changed of
> course. The easiest step 1 is to just use gzopen() from zlib on the
> files and use the same code as now :-)

Oh if integrating it is easier :)

>> Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
>> not already in SHM?
>
> I think that's way more complex than we want to go here.

Yeah.

>> Well, I would guess that if you're streaming the WAL files in parallel
>> while the base backup is taken, then you're able to have it all without
>> archiving setup, and the server could still recycling them.
>
> Yes, this was mostly for the use-case of "getting a single tarfile
> that you can actually use to restore from without needing the log
> archive at all".

It also allows for a simpler kick-start procedure for preparing a
standby, and allows to stop worrying too much about wal_keep_segments
and archive servers.

When do the standby launch its walreceiver? It would be extra-nice for
the base backup tool to optionally continue streaming WALs until the
standby starts doing it itself, so that wal_keep_segments is really
deprecated. No idea how feasible that is, though.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support