HELP: How to tame the 8.3.x JDBC driver with a biq guery result set

From: Dave Crooke <dcrooke(at)gmail(dot)com>
To: pgsql-performance <pgsql-performance(at)postgresql(dot)org>
Subject: HELP: How to tame the 8.3.x JDBC driver with a biq guery result set
Date: 2010-04-15 19:42:51
Message-ID: u2kca24673e1004151242k1fa7e2a0zcc0e4f4fbec5eb62@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

Hey folks

I am trying to do a full table scan on a large table from Java, using a
straightforward "select * from foo". I've run into these problems:

1. By default, the PG JDBC driver attempts to suck the entire result set
into RAM, resulting in *java.lang.OutOfMemoryError* ... this is not cool, in
fact I consider it a serious bug (even MySQL gets this right ;-) I am only
testing with a 9GB result set, but production needs to scale to 200GB or
more, so throwing hardware at is is not feasible.

2. I tried using the official taming method, namely *
java.sql.Statement.setFetchSize(1000)* and this makes it blow up entirely
with an error I have no context for, as follows (the number C_10 varies,
e.g. C_12 last time) ...

org.postgresql.util.PSQLException: ERROR: portal "C_10" does not exist
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1592)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1327)
at
org.postgresql.core.v3.QueryExecutorImpl.fetch(QueryExecutorImpl.java:1527)
at
org.postgresql.jdbc2.AbstractJdbc2ResultSet.next(AbstractJdbc2ResultSet.java:1843)

This is definitely a bug :-)

Is there a known workaround for this ... will updating to a newer version of
the driver fix this?

Is there a magic incation of JDBC calls that will tame it?

Can I cast the objects to PG specific types and access a hidden API to turn
off this behaviour?

If the only workaround is to explicitly create a cursor in PG, is there a
good example of how to do this from Java?

Cheers
Dave

Responses

Browse pgsql-performance by date

  From Date Subject
Next Message Josh Berkus 2010-04-15 19:44:31 Autovaccum with cost_delay does not complete on one solaris 5.10 machine
Previous Message Kevin Grittner 2010-04-15 17:47:26 Re: 8.3.9 - latency spikes with Linux (and tuning for consistently low latency)