Re: [FEATURE REQUEST] Streaming Onlinebackup (MaybeOFFTOPIC)

On Thu, 2007-09-27 at 10:17 +0200, Zeugswetter Andreas ADI SD wrote:
> > > Attached is a modified version to implement both of these. I also
> bailed
> > > out if there was surplus input. I tried an optimization of
> allocating a
> > > separate buffer for outputting the zeros, to avoid repeated memset
> calls.
> > > It didn't seem to make a very big difference; do you think it's
> worth
> > > cluttering the code with that?
> >
> > Would it work to just ftruncate the file?
>
> We would need to teach recovery to accept a short file if the last
> record is a
> valid switch log XLOG record. RestoreArchivedFile currently bails out if
> the file
> size is not XLogSegSize.

We also need to check in pg_standby to see whether the file is still
being copied or is complete. Currently, we check the filesize and wait
for it to be 16M. There would need to be another way for pg_standby to
assess whether the file has completed transfer before copying into the
data directory of the standby node.

> We need to make exact checks though, or this would reduce reliability.
> (e.g. a short file must have records up to the very end)
>
> The probably useful next step would be to pass the current length to the
> archive_command,
> so it can write the filled part of the file without the need for a
> filter.

It's certainly possible to pass file metadata as well as the file.

I'd be worried that would significantly complicate the mechanism, which
at the moment is clean and simple; so this isn't gonna happen for 8.3
AFAICS. Changing that would require people to redesign their HA configs,
which doesn't sound that great to me.

The next step for me is to stream the records, not to fuss too much with
the existing file level copying. Streaming provides what we really want:
a shorter delay in data transfer between primary and standby, without
additional overhead.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com