Skip site navigation (1) Skip section navigation (2)

Re: PLPythonU & Out of Memory - Importing Query

From: Michael Fuhr <mike(at)fuhr(dot)org>
To: Jon Clements <jon(dot)clements(at)trgstrata(dot)co(dot)uk>
Cc: "pgsql-interfaces(at)postgresql(dot)org" <pgsql-interfaces(at)postgresql(dot)org>
Subject: Re: PLPythonU & Out of Memory - Importing Query
Date: 2005-11-09 18:05:05
Message-ID: 20051109180505.GA41806@winnie.fuhr.org (view raw or flat)
Thread:
Lists: pgsql-interfaces
On Tue, Nov 08, 2005 at 02:21:01PM +0000, Jon Clements wrote:
> I am currently experimenting using plpythonu with postgresql 8.0
> for Win32. It's basically a quick script that imports data from CSV
> files, but does some quite complicated data lookups and selections.
> The area in which I'm somewhat confunded is memory usage. The process
> successfully runs, but keeps climbing in memory usage relentlessly,
> successfully importing about 200k records, before the memory usage
> of postgres soars to 2gb and of course, shortly after that, grinds
> to a halt with a "Out of Memory" error.
[ ... ]
> # Setup external CSV data source
> # For each record, that meets certain critera, execute insert...
> for rec in dsource: plpy.execute(myplan, [Value1, Value2] )

What exactly is dsource?  If it were a simple list that was built
in its entirety prior to the "for" loop then I'd expect that to
cause memory problems; however, since you're getting a partial data
load I'm wondering if dsource is some other kind of object that
implements an iterator.  Have you run the same code (minus the
database parts) in a standalone Python program to see if it has a
memory leak?

-- 
Michael Fuhr

In response to

pgsql-interfaces by date

Next:From: Scott LambDate: 2005-11-09 19:36:29
Subject: Incremental results from libpq
Previous:From: Tom LaneDate: 2005-11-09 16:52:35
Subject: Re: PLPythonU & Out of Memory - Importing Query

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group