Table partitioning for maximum speed?

From: Jeff Boes <jboes(at)nexcerpt(dot)com>
To: pgsql-general(at)postgresql(dot)org
Subject: Table partitioning for maximum speed?
Date: 2003-10-09 18:37:19
Message-ID: 7f84678fe08ecf31a76aa0dda09fd8f2@news.teranews.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

I'm sure this is a concept that's been explored here. I have a table
(fairly simple, just two columns, one of which is a 32-digit checksum)
with several million rows (currently, about 7 million). About a million
times a day we do

select * from my_table where md5 = ?

to verify presence or absence of the row, and base further processing on
that information.

The idea bandied about now is to partition this table into 16 (or 256,
or ...) chunks by first digit (or 2, or ...). In the simplest case, this
would mean:

create table my_table_0 as select * from my_table where md5 like '0%';

create table my_table_1 as select * from my_table where md5 like '1%';

...

create table my_table_f as select * from my_table where md5 like 'f%';

Then change the code to examine the checksum and create a query to the
appropriate table based on the first digit.

Obviously, this is conceptually similar to what the index on the "md5"
column is supposed to do for us. However, partitioning moves just a
little of the processing load off the database server and onto the
machine running the application. That's important, because we can afford
more application machines as load increases, but we can't as easily
upgrade the database server.

Will a query against a table of 0.5 million rows beat a query against a
table of 7 million rows by a margin that makes it worth the hassle of
supporting 15 "extra" tables?

--
Jeff Boes vox 269.226.9550 ext 24
Database Engineer fax 269.349.9076
Nexcerpt, Inc. http://www.nexcerpt.com
...Nexcerpt... Extend your Expertise

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Bruno Wolff III 2003-10-09 18:43:17 Re: Parent Id
Previous Message Christopher Browne 2003-10-09 18:33:42 Re: Response from MySql AB (Re: Humor me: Postgresql vs.