Skip site navigation (1) Skip section navigation (2)

Re: optimized counting of web statistics

From: Rudi Starcevic <tech(at)wildcash(dot)com>
To: Postgresql Performance list <pgsql-performance(at)postgresql(dot)org>
Subject: Re: optimized counting of web statistics
Date: 2005-06-29 17:17:41
Message-ID: 42C2D7B5.4070707@wildcash.com (view raw or flat)
Thread:
Lists: pgsql-performance
Hi,

>I do my batch processing daily using a python script I've written. I
>found that trying to do it with pl/pgsql took more than 24 hours to
>process 24 hours worth of logs. I then used C# and in memory hash
>tables to drop the time to 2 hours, but I couldn't get mono installed
>on some of my older servers. Python proved the fastest and I can
>process 24 hours worth of logs in about 15 minutes. Common reports run
>in < 1 sec and custom reports run in < 15 seconds (usually).
>  
>

When you say you do your batch processing in a Python script do you mean
a you are using 'plpython' inside
PostgreSQL or using Python to execut select statements and crunch the
data 'outside' PostgreSQL?

Your reply is very interesting.

Thanks.
Regards,
Rudi.


In response to

Responses

pgsql-performance by date

Next:From: Martin LesserDate: 2005-06-30 07:24:06
Subject: Vacuum becomes slow
Previous:From: Tom LaneDate: 2005-06-29 15:27:47
Subject: Re: Exclusive lock question

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group