Re: suggestions to improve postgresql suitability for data-mining

From: "Darren King" <DarrenK(at)Routescape(dot)com>
To: "PostgreSQL Developers" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: suggestions to improve postgresql suitability for data-mining
Date: 2003-07-22 17:52:20
Message-ID: 26751392596DDD4D84FE1806D1F819B794A963@exchange.insight
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers


> You want to process all invoices to count them
> and to sum up the amounts on a per month/area/type
> basis. The initial data size is in GB, but the
> size of the expected result is in KB (namely 2 data
> for each 100 areas * 12 months * 4 types).

The key to handling large datasets for data mining is pre-aggregation based on the smallest time frame needed for details.

I'd suggest running these large queries and storing the results in other tables, and then writing a set of functions to work with those aggregate tables.

No sense in summing up the same set of static data more than once if you can help it.

Darren

Browse pgsql-hackers by date

  From Date Subject
Next Message Rod Taylor 2003-07-22 17:57:37 Re: php with postgres
Previous Message Bruno Wolff III 2003-07-22 17:43:58 Re: suggestions to improve postgresql suitability for data-mining