Re: optimized counting of web statistics

From: Rudi Starcevic <tech(at)wildcash(dot)com>
To: Postgresql Performance list <pgsql-performance(at)postgresql(dot)org>
Subject: Re: optimized counting of web statistics
Date: 2005-06-29 17:17:41
Message-ID: 42C2D7B5.4070707@wildcash.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

Hi,

>I do my batch processing daily using a python script I've written. I
>found that trying to do it with pl/pgsql took more than 24 hours to
>process 24 hours worth of logs. I then used C# and in memory hash
>tables to drop the time to 2 hours, but I couldn't get mono installed
>on some of my older servers. Python proved the fastest and I can
>process 24 hours worth of logs in about 15 minutes. Common reports run
>in < 1 sec and custom reports run in < 15 seconds (usually).
>
>

When you say you do your batch processing in a Python script do you mean
a you are using 'plpython' inside
PostgreSQL or using Python to execut select statements and crunch the
data 'outside' PostgreSQL?

Your reply is very interesting.

Thanks.
Regards,
Rudi.

In response to

Responses

Browse pgsql-performance by date

  From Date Subject
Next Message Martin Lesser 2005-06-30 07:24:06 Vacuum becomes slow
Previous Message Tom Lane 2005-06-29 15:27:47 Re: Exclusive lock question