Re: Streaming replication, and walsender during recovery

Simon Riggs wrote:
> On Thu, 2010-01-28 at 10:48 -0500, Tom Lane wrote:
>> Fujii Masao <masao(dot)fujii(at)gmail(dot)com> writes:
>>> How about just making a restore_command copy the WAL files as the
>>> normal one (e.g., 0000...) instead of a pg_xlog/RECOVERYXLOG?
>>> Though we need to worry about deleting them, we can easily leave
>>> the task to the bgwriter.
>> The reason for doing it that way was to limit disk space usage during
>> a long restore. I'm not convinced we can leave the task to the bgwriter
>> --- it shouldn't be deleting anything at that point.
>
> I think "bgwriter" means RemoveOldXlogFiles(), which would normally
> clear down files at checkpoint. If that was added to the end of
> RecoveryRestartPoint() to do roughly the same job then it could
> potentially work.

SR added a RemoveOldXLogFiles() call to CreateRestartPoint().

(Since 8.4, RecoveryRestartPoint() just writes the location of the
checkpoint record in shared memory, but doesn't actually perform the
restartpoint; bgwriter does that in CreateRestartPoint()).

> However, since not every checkpoint is a restartpoint we might easily
> end up with significantly more WAL files on the standby than would
> normally be there when it would be a primary. Not sure if that is an
> issue in this case, but we can't just assume we can store all files
> needed to restart the standby on the standby itself, in all cases. That
> might be an argument to add a restartpoint_segments parameter, so we can
> trigger restartpoints on WAL volume as well as time. But even that would
> not put an absolute limit on the number of WAL files.

I think it is a pretty important safety feature that we keep all the WAL
around that's needed to recover the standby. To avoid out-of-disk-space
situation, it's probably enough in practice to set checkpoint_timeout
small enough in the standby to trigger restartpoints often enough.

At the moment, we do retain streamed WAL as long as it's needed, but not
the WAL restored from archive.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com