Re: storing large amounts of text

Just to give everyone here a head's up as to what's coming: The developers
have long known that the size limit on tuple storage (particularly as it
affects the 'text' type) needs to go away, and that the existing large
objects (lo) are less than ideal as a work around. Jan Wieck has done
the inital plan and development of a generic replacement: the Oversized
Tuple Storage Technique (TOAST).

http://www.postgresql.org/projects/devel-toast.html

This system allows automatic compression and/or external storage of
any datatype, directly in the database. It's still in development, but
Jan has recently released a snapshot, so it's past the initial planning
stages. For now, we'll all have to make do with lo or external file store,
but there is hope on the horizon...

Ross
--
Ross J. Reedstrom, Ph.D., <reedstrm(at)rice(dot)edu>
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St., Houston, TX 77005

On Sat, Apr 22, 2000 at 03:32:32PM -0700, Titus Brown wrote:
> -> I'm sure this has come up before because it just logically seems like
> -> others would have run up against this too. Is it possible to store a
> -> large amount of text in a row/field? I want to store articles in the
> -> database along with information about them, sort of a content databasing
> -> system. Can anyone reccommend a good way to go about doing this in
> -> Postgres? Any and all help greatly appreciated.
>
> I'm doing this for a genome project; you need to look into large object
> functionality if you want to store anything larger than 8kb in PostgreSQL.
>
> This is pretty inconvenient to do, because it's slow to search (I had
> to write my own search routine; it's at
>
> http://www.idyll.org/~t/www-tools/
>
> along with some other stuff for AOLserver/large objects) and you have
> to use a special program to dump the large objects (again, contact me,
> or refer to the archives).
>
> Several people have recommended that you store the files themselves on
> your file system and have just the indices be kept in the database. That
> should work as well, and I'll actually be looking into it today.
>
> cheers,
> --titus