Re: lots of large objects and toast

Tom Lane wrote:
> Lincoln Yeoh <lylyeoh(at)mecomb(dot)com> writes:
> >> There's never been much enthusiasm among the core developers for large
> >> objects at all --- we see them as a poor substitute for allowing large
> >> values directly. (The "TOAST" work scheduled for 7.1 will finally
> >> resolve that issue, I hope.) So no one's felt like working on improving
> >> the large-object implementation.
>
> > On the practical side, say I want to insert/read a large amount of
> > information into/from a TOAST field. How should I do it?
> > Is there a pipe method where I can continuously print to/read from?
>
> Not at the moment, but that's obviously going to be a necessary feature
> if we want to make the existing flavor of large objects obsolete. There
> have been some preliminary discussions about it --- AFAIR no one's laid
> out a complete proposal yet.

Yes, we already saw that problem. And looking at some other
databases they seem to deal with it the same way I'm inclined
to do.

The BLOB/CLOB data types have to be references. A dummy
reference is created by a special function that's used in the
values for INSERT. If such a dummy ref occurs in a heap
tuple to be stored, a real empty reference is created.
Wherever we store the content, we need some new interface in
libpq to access them like files with seek/read/write.

It's somehow like our current LO, but all LOs for one column
reside in one shadow table, and the system definitely knows
when an item is obsolete.

The type output functions just output the reference, and new
libpq functions then gain access to it. It will need some
enhancements on the SQL level too, to make pg_dump able to
save/restore them. The shadow tables might have a different
relkind, so they aren't accessible via normal SQL. But if
COPY can, why not?

All that is still a little vague and there are problems I
don't want to talk about right now. I need to get TOAST
ready for 7.1, then go back to FOREIGN KEY and implement the
file buffered trigger queue. So I don't expect BLOB/CLOB to
be ready before 7.2.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #