Re: PITR Archival

On Tue, 2004-06-15 at 16:34, Simon Riggs wrote:
> I enclose a working set of context diff patches and new files to make
> PITR archiving work, as of cvstip (NOW).

As of now, this patch is invalidated by recent changes. Don't try to run
it, it doesn't even start. There's good news coming later in this email
update.... :)

I'm now in the middle of reworking this, 1-3 days, I guess.

> There is one bug: shutdown doesn't work quite right. I haven't fixed
> this because I've spent too long trying to decipher how pgstat did a
> clean shutdown, discovering now that it didn't and that has now been
> patched...something similar is required for pgarch, but I'm out of time
> now...leaving time for discussion of this lot...
>

The code that doesn't work appears to be the Archiver startup code, so
that effectively requires me to rework the startup/shutdown code that
was slightly flaky, as mentioned above.

I originally cloned the pgstat code, which as I observed was possible to
improve upon....that having been done (a good thing) should just leave
the task as a re-clone (especially since that par of the code I can
almost recite by now).

I would welcome one of the Win32 crew having an eye through the code, to
suggest any areas to avoid/pay close attention to. I don't want to
invalidate any other work, seeing as I'm last to show now at this party.

> If we get this done smoothly, I reckon I can have some PITR recovery
> control done by beta freeze.

The good news: ...before recent changes, I had a working version of the
streaming recovery code and recovered 3 test databases. I would note
that the recovery speed was quite impressive, even on my 700Mhz dev pc,
so I have confidence that this will work well on the real deal.

How this works: when startup enters recovery, if you have supplied a
file: DataDir/recovery.conf, then it will read this to get a recovery
command line. startup executes this command each time recovery requests
a new xlog from archive, using only one file at a time.
Result: you will be able to recover, no matter how many xlogs you have
and how little disk space you have (in comparison).

This will allow you to write very short scripts to perform
- "infinite recovery", just like Oracle 9i and DB2 8.1
- integration with tapes and backup software
- automated standby databases, where an active database feeds logs to a
passive database that is permanently "InRecovery", yet can be brought
on-line within a few seconds if the link drops.

I haven't yet written the code to stop at-a-point-in-time, but I remain
ever optimistic. I won't slow down the patch waiting for that...

...and I take it the doc freeze is not the same time as the code
freeze..

Best regards, Simon Riggs