Re: Page-level version upgrade (was: Block-level CRC checks)

On Tue, Dec 1, 2009 at 10:45 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> Robert Haas wrote:
>>
>> If the pre-upgrade utility is something
>> that has to be run while the database is off-line, then it defeats the
>> point of an in-place upgrade.  If it can be run while the database is
>> up, I fear it will need to be deeply integrated into the server.  And
>> since we can't know the requirements for how much space to reserve
>> (and it needn't be a constant) until we design the new feature, this
>> will likely mean backpatching a rather large chunk of complex code,
>> which to put it mildly, is not the sort of thing we normally would
>> even consider.
>
> You're wandering into the sort of overdesign that isn't really needed yet.
>  For now, presume it's a constant amount of overhead, and that the release
> notes for the new version will say "configure the pre-upgrade utility and
> tell it you need <x> bytes of space reserved".  That's sufficient for the
> CRC case, right?  Needs a few more bytes per page, 8.5 release notes could
> say exactly how much.  Solve that before making things more complicated by
> presuming you need to solve the variable-size increase problem, too.  We'll
> be lucky to get the absolute simplest approach committed, you really need to
> have a big smoking gun to justify feature creep in this area.

Well, I think the best way to solve the problem is to design the
system in a way that makes it unnecessary to have a pre-upgrade tool
at all, by making the new PG version capable of handling page
expansion where needed. I don't understand how putting that
functionality into the OLD PG version can be better. But I may be
misunderstanding something.

> (If I had to shoot from the hip and design for the variable case, why not
> just make the thing that determines how much space a given page needs
> reserved a function the user can re-install with a smarter version?)

That's a pretty good idea. I have no love of this pre-upgrade
concept, but if we're going to do it that way, then allowing someone
to load in a function to compute the required amount of free space to
reserve is a good thought.

>> I think a better approach is to support reading tuples
>> from old pages, but to write all new tuples into new pages.  A
>> full-table rewrite (like UPDATE foo SET x = x, CLUSTER, etc.) can be
>> used to propel everything to the new version, with the usual tricks
>> for people who need to rewrite the table a piece at a time.
>
> I think you're oversimplifying the operational difficulty of "the usual
> tricks".  This is a painful approach for the exact people who need this the
> most:  people with a live multi-TB installation they can't really afford to
> add too much load to.  The beauty of the in-place upgrade tool just
> converting pages as it scans through looking for them is that you can dial
> up its intensity to exactly how much overhead you can stand, and let it
> loose until it's done.

Fair enough.

...Robert