Re: Pie-in-sky dreaming about reworking tuple layout entirely

On Oct 3, 2006, at 4:06 PM, Merlin Moncure wrote:
> On 10/3/06, Gregory Stark <stark(at)enterprisedb(dot)com> wrote:
>> I can't shake the feeling that merely tweaking the way our
>> varlenas work with
>> a shortvarlena or with compressed varlena headers is missing the
>> real source
>> of our headaches. It seems very strange to me to be trying to step
>> through a
>> tuple with length bits at the head of every field. It's a lot of
>> work spent
>> dealing with a terribly inconvenient format when we can pick the
>> format to be
>> whatever we like.
>
> one advantage of the current system is that columns with nulls do not
> require any storage. so you can alter table add column for free on a
> really big table. ISTM that your approch would require moving all the
> static fields in if you added a static field regardless, right?

I'm thinking that for Greg's ideas to be workable, we'll need to
divorce the on-disk format from what was specified in CREATE TABLE,
specifically so we can do things like put all the fixed-width stuff
in front of the variable-width stuff (of course we could also further
optimize placement beyond that).

IIRC, the show-stopper for doing that is how to deal with DDL
changes. While we could try and get cute about that, there is a brute-
force method that would work without a doubt: store some kind of
catalog version number in each tuple (or maybe just in each page,
since you could theoretically convert an entire page to a different
format without too big a penalty while you've already got it in memory.

There are some caveats to this... there will be some limit on how
many DDL changes you can make until you run out of version numbers.
Worst-case, we could provide a command that would run through the
entire table, ensuring that everything is up to the current version.
Of course, we'd want some way to throttle that, but I don't think
that'd be terribly difficult. One nice thing is that you shouldn't
need to mess with any visibility info when you run this, so it should
be able to just do everything in-place.

BTW, it seems like what we're really looking at between this and
discussion of visibility changes, etc. is essentially re-designing
the entire storage layout (for better or for worse).
--
Jim Nasby jimn(at)enterprisedb(dot)com
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)