Re: Variable length varlena headers redux

Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> That seems like an awful lot of copying and pallocs that aren't there
> currently though. And it'll make us reluctant to change over frequently used
> data types like text -- which are precisely the ones that would gain us the
> most.

> It seems to me that it might be better to change to storing varlena lengths in
> network byte order instead. That way we can dedicate the leading bits to toast
> flags and read more bytes as necessary.

This'll add its own overhead ... but probably less than pallocs and
data-copying would. And I agree we can find (pretty much) all the
places that need changing by the expedient of deliberately renaming
the macros and struct fields.

One problem I foresee is that I think you are about to propose that
VARDATA depend on the length already having been inserted, which it
does not now; and simple renamings won't detect ordering errors for
that. Also I believe there are places that over-allocate memory,
fill in the data, and only then set the length; something you will
not easily be able to change. It might work if we assume that
*creation* of a varlena value always produces the 4-byte-header form
and only reading of a value that might be on disk needs to cope with
the short-header forms. However this seems to require two different
forms of VARDATA depending on whether one is preparing or reading a
value. Ugh.

regards, tom lane