Re: update on TOAST status'

Tom Lane wrote:
> JanWieck(at)t-online(dot)de (Jan Wieck) writes:
> > After debugging something I thought was a bug in the toaster,
> > I've found something really causing headaches.
> > TOAST AS IS IS NOT CAPABLE OF HOLDING INDEXED VALUES!
> > It appears that brtee indices (at least) can keep references
> > to old toast values that survive a VACUUM! Seems these
> > references live in nodes actually not referring to a heap
> > tuple any more, but used during tree traversal in
> > comparisions. As if an index tuple delete from a btree not
> > necessarily causes the index value to disappear from the
> > btree completely. It'll never be returned by an index scan,
> > but the value is still there somewhere.
>
> Oooh, nasty. Probably the keys you are looking at are in upper-
> level btree pages and indicate the ranges of keys found in lower
> pages, rather than being pointers to real tuples.

So our btree implementation is closer to an ISAM file
organization than to a real tree? Anyway, either one or the
other is the reason that an attempt to insert a new value
results in an lztext_cmp() call that cannot be resolved due
to a missing toast value.

I added some checks to the detoaster just to throw an
elog(ERROR) instead of a coredump in such a case earlier
today.

> One answer is to rebuild indexes from scratch during VACUUM,
> before we vacuum the TOAST relation. We've been talking about
> doing that for a long time. Maybe it's time to bite the bullet
> and do it. (Of course that means fixing the relation-versioning
> problem, which it seems we don't have a consensus on yet...)

Doesn't matter if we do it before or after, because the main
heap shouldn't contain any more toast references to deleted
(later to be vacuumed) toast entries at that time.

Anyway, it's a nice idea that should solve the problem. For
indices, which can allways be rebuilt from the heap data, I
don't see such a big need for the versioning. Only that a
partially rebuilt index (rebuild crashed in the middle) needs
another vacuum before the the DB is accessible again. How
often does that happen?

So why not having vacuum truncating the index file to zero
and rebuilding it from scratch in place? Can anyone access an
index while vacuum has a lock on it's heap?

>
> > Seems the designs of btree and toast are colliding. As soon
> > as "text" is toastable, this'll hurt - be warned.
>
> Text *is* marked toastable in current CVS...

Whow - haven't noticed.

Will run my tests against text ... parallel. Does it have any
impact on the regression test execution time? Does any toast
table (that should now be there in the regression DB) loose
it's zero size during the tests?

>
> > AFAICS, we need to detoast values for index inserts allways
> > and have another toaster inside the index access methods at
> > some day. In the meantime we should decide a safe maximum
> > index tuple size and emit an explanative error message on the
> > attempt to insert oversized index entries instead of possibly
> > corrupting the index.
>
> I don't like that --- seems it would put a definite crimp in the
> whole point of TOAST, which is not to have arbitrary limits on field
> sizes.

If we can solve it, let's do so. If we cannot, let's restrict
it for 7.1.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #