Something's been bugging me

A while back in an off-hand comment Tom packed varlenas he mentioned that we
might want to have more types of toast pointers. Since then the idea of some
alternative column-wise partitioning scheme has come up and another idea I've
been tossing around is some kind of compression scheme which takes advantage
of information across rows.

The current varvarlena scheme doesn't leave any room for any expansion at all.
Every possible value of varlena headers is meaningful and could potentially
exist in a database. So any such scheme would be end any hope of binary
compatibility in the future and probably any hope of a migrator since it would
potentially expand existing data.

Now not every possible idea for these options would need space in the varlena
header but many would.

I'm wondering whether it doesn't make sense to lower VARATT_SHORT_MAX to 0x70
to allow for at least a small number of constant values which could indicate
some special type of datum. That could be used to indicate that a fixed size
pointer like a toast pointer follows. That could be used for something like
common value compression. [*]

I'm almost tempted to suggest lowering it as far as 0x3f which would give us a
whole bit. That would be necessary if we wanted to allow for variable length
fields with some alternate interpretation following the header such as some
very light compression scheme. But I'm pretty loath to give up as much as that
now for only a potential future gain.

[*] Yes, this does make the idea of keeping the VARHDRSZ/VARSHDRSZ offset in
the varlen header seem pretty silly; hindsight is 20/20 and all that.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com