Re: Out of memory error on huge resultset

From: Aaron Mulder <ammulder(at)alumni(dot)princeton(dot)edu>
To: pgsql-jdbc <pgsql-jdbc(at)postgresql(dot)org>
Subject: Re: Out of memory error on huge resultset
Date: 2002-10-11 14:30:49
Message-ID: Pine.LNX.4.44.0210111029160.695-100000@www.princetongames.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general pgsql-hackers pgsql-jdbc

What would be the disadvantage of making the JDBC driver use a
cursor under the covers (always)? Is it significantly slower or more
resource-intensive than fetching all the data at once? Certainly it seems
like it would save memory in some cases.

Aaron

On 10 Oct 2002, Dave Cramer wrote:
> Nick,
>
> Use a cursor, the current driver doesn't support caching, the backend
> gives you everything you ask for, you can't just say you want a limited
> set.
>
> So if you use cursors you can fetch a subset
>
> Dave
> On Thu, 2002-10-10 at 11:24, Nick Fankhauser wrote:
> >
> > I'm selecting a huge ResultSet from our database- about one million rows,
> > with one of the fields being varchar(500). I get an out of memory error from
> > java.
> >
> > If the whole ResultSet gets stashed in memory, this isn't really surprising,
> > but I'm wondering why this happens (if it does), rather than a subset around
> > the current record being cached and other rows being retrieved as needed.
> >
> > If it turns out that there are good reasons for it to all be in memory, then
> > my question is whether there is a better approach that people typically use
> > in this situation. For now, I'm simply breaking up the select into smaller
> > chunks, but that approach won't be satisfactory in the long run.
> >
> > Thanks
> >
> > -Nick
> >
> > --------------------------------------------------------------------------
> > Nick Fankhauser nickf(at)ontko(dot)com Phone 1.765.935.4283 Fax 1.765.962.9788
> > Ray Ontko & Co. Software Consulting Services http://www.ontko.com/
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 5: Have you checked our extensive FAQ?
> >
> > http://www.postgresql.org/users-lounge/docs/faq.html
> >
> >
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

In response to

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Greg Copeland 2002-10-11 14:42:12 Re: MySQL vs PostgreSQL.
Previous Message Dave Cramer 2002-10-11 14:30:30 Re: Out of memory error on huge resultset

Browse pgsql-hackers by date

  From Date Subject
Next Message Greg Copeland 2002-10-11 14:42:12 Re: MySQL vs PostgreSQL.
Previous Message Dave Cramer 2002-10-11 14:30:30 Re: Out of memory error on huge resultset

Browse pgsql-jdbc by date

  From Date Subject
Next Message Greg Copeland 2002-10-11 14:42:12 Re: MySQL vs PostgreSQL.
Previous Message Dave Cramer 2002-10-11 14:30:30 Re: Out of memory error on huge resultset