Skip site navigation (1) Skip section navigation (2)

Re: Significantly larger toast tables on 8.4?

From: "Stephen R(dot) van den Berg" <srb(at)cuci(dot)nl>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Alex Hunsaker <badalex(at)gmail(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>,PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Significantly larger toast tables on 8.4?
Date: 2009-01-02 21:19:00
Message-ID: 20090102211900.GB18016@cuci.nl (view raw or flat)
Thread:
Lists: pgsql-hackers
Robert Haas wrote:
>On Fri, Jan 2, 2009 at 3:23 PM, Stephen R. van den Berg <srb(at)cuci(dot)nl> wrote:
>> Three things:
>> a. Shouldn't it in theory be possible to have a decompression algorithm
>>   which is IO-bound because it decompresses faster than the disk can
>>   supply the data?  (On common current hardware).
>> b. Has the current algorithm been carefully benchmarked and/or optimised
>>   and/or chosen to fit the IO-bound target as close as possible?
>> c. Are there any well-known pitfalls/objections which would prevent me from
>>   changing the algorithm to something more efficient (read: IO-bound)?

>Any compression algorithm is going to require you to decompress the
>entire string before extracting a substring at a given offset.  When
>the data is uncompressed, you can jump directly to the offset you want
>to read.  Even if the compression algorithm requires no overhead at
>all, it's going to make the location of the data nondeterministic, and
>therefore force additional disk reads.

That shouldn't be insurmountable:
- I currently have difficulty imagining applications that actually do
  lots of substring extractions from large compressible fields.
  The most likely operation would be a table which contains tsearch
  indexed large textfields, but those are unlikely to participate in
  a lot of substring extractions.
- Even if substring operations would be likely, I could envision a compressed
  format which compresses in compressed chunks of say 64KB which can then
  be addressed randomly independently.
-- 
Sincerely,
           Stephen R. van den Berg.

"Always remember that you are unique.  Just like everyone else."

In response to

Responses

pgsql-hackers by date

Next:From: Aidan Van DykDate: 2009-01-02 21:33:34
Subject: Re: Several tags around PostgreSQL 7.1 broken
Previous:From: Gregory StarkDate: 2009-01-02 20:59:35
Subject: Re: Significantly larger toast tables on 8.4?

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group