Re: extensible external toast tuple support

On 2013-06-28 09:49:53 -0400, Robert Haas wrote:
> On Fri, Jun 28, 2013 at 9:23 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> > On Thu, Jun 27, 2013 at 12:56 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> >> Please find attached the next version of the extensible toast
> >> support. There basically are two changes:
> >>
> >> * handle indirect toast tuples properly in heap_tuple_fetch_attr
> >> and related places
> >> * minor comment adjustments
> >
> > It looks to me like you need to pass true, rather than false, as the
> > first argument to TrapMacro:
> >
> > +#define VARTAG_SIZE(tag) \
> > + ((tag) == VARTAG_INDIRECT ? sizeof(varatt_indirect) : \
> > + (tag) == VARTAG_ONDISK ? sizeof(varatt_external) : \
> > + TrapMacro(false, "unknown vartag"))

Heh. I was obviously trying to be too cute ;)

Good idea & thanks for committing it separately.

> Do you see external-indirect values getting used for anything other
> than logical replication? Is there code to do so anywhere?

Yes and not really. I think we can reuse it to avoid repeated detoastings, even
in relatively normal queries. What I am absolutely not sure yet is how
to automatically decide when we want to keep the tuple in memory because
it will be beneficial, and when not because of the obvious memory
overhead. And how to keep track of the memory context used to allocate
the referenced data.
There are some usecases where I think it might be relatively easy to
decide its a win. E.g tuples passed to triggers if there are
multiple ones of them.

I've played a bit with using that functionality in C functions, but
nothing publishable, more to make sure things work.

> Why does toast_insert_or_update() need to go through all the
> rigamarole in toast_datum_differs()? I would have thought that it
> could simply treat any external-indirect value as needing to be
> detoasted and retoasted, since the destination is the disk anyhow.

We could do that, yes. But I think it might be better not to: If we
simplify the tuples used in a query to not reference ondisk tuples
anymore and we then UPDATE using that new version I would rather not
retoast all the unchanged columns.

I can e.g. very well imagine that we decide to resolve toasted Datums to
indirect Datums during an UPDATE if there are multiple BEFORE UPDATE
triggers to avoid detoasting in each and every one of them. Such a tuple
will then passed to heap_update...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services