Skip site navigation (1) Skip section navigation (2)

Re: Triggers During COPY

From: Mark Kirkwood <markir(at)coretech(dot)co(dot)nz>
To: "Thomas F(dot)O'Connell" <tfo(at)sitening(dot)com>
Cc: PgSQL - Performance <pgsql-performance(at)postgresql(dot)org>
Subject: Re: Triggers During COPY
Date: 2005-01-28 05:55:37
Message-ID: 41F9D3D9.40909@coretech.co.nz (view raw or flat)
Thread:
Lists: pgsql-performance
Thomas F.O'Connell wrote:
> 
> The problem comes in importing new data into the tables for which the 
> counts are maintained. The current import process does some 
> preprocessing and then does a COPY from the filesystem to one of the 
> tables on which counts are maintained. This means that for each row 
> being inserted by COPY, a trigger is fired. This didn't seem like a big 
> deal to me until testing began on realistic data sets.
> 
> For a 5,000-record import, preprocessing plus the COPY took about 5 
> minutes. Once the triggers used for maintaining the counts were added, 
> this grew to 25 minutes. While I knew there would be a slowdown per row 
> affected, I expected something closer to 2x than to 5x.
> rformance out of this scenario?
> 
Have been seeing similar behavior whilst testing sample code for the 8.0
docs (summary table plpgsql trigger example).

I think the nub of the problem is dead tuples bloat in the summary /
count table, so each additional triggered update becomes more and more
expensive as time goes on. I suspect the performance decrease is
exponential with the no of rows to be processed.


> Would it be absurd to drop the triggers during import and recreate them 
> afterward and update the counts in a summary update based on information 
> from the import process?
> 
> 
That's the conclusion I came to :-)

regards

Mark


In response to

pgsql-performance by date

Next:From: Thomas F.O'ConnellDate: 2005-01-28 06:28:58
Subject: Re: Triggers During COPY
Previous:From: Josh BerkusDate: 2005-01-28 05:41:49
Subject: Re: Triggers During COPY

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group