Skip site navigation (1) Skip section navigation (2)

Re: optimized counting of web statistics

From: Rudi Starcevic <tech(at)wildcash(dot)com>
To: Postgresql Performance list <pgsql-performance(at)postgresql(dot)org>
Subject: Re: optimized counting of web statistics
Date: 2005-06-29 17:17:41
Message-ID: (view raw or whole thread)
Lists: pgsql-performance

>I do my batch processing daily using a python script I've written. I
>found that trying to do it with pl/pgsql took more than 24 hours to
>process 24 hours worth of logs. I then used C# and in memory hash
>tables to drop the time to 2 hours, but I couldn't get mono installed
>on some of my older servers. Python proved the fastest and I can
>process 24 hours worth of logs in about 15 minutes. Common reports run
>in < 1 sec and custom reports run in < 15 seconds (usually).

When you say you do your batch processing in a Python script do you mean
a you are using 'plpython' inside
PostgreSQL or using Python to execut select statements and crunch the
data 'outside' PostgreSQL?

Your reply is very interesting.


In response to


pgsql-performance by date

Next:From: Martin LesserDate: 2005-06-30 07:24:06
Subject: Vacuum becomes slow
Previous:From: Tom LaneDate: 2005-06-29 15:27:47
Subject: Re: Exclusive lock question

Privacy Policy | About PostgreSQL
Copyright © 1996-2015 The PostgreSQL Global Development Group