We are having a lot of trouble trying to use JDBC with PostgreSQL.
This is what we want to achieve:
- Select all rows in a table (people)
- Select addresses belonging to each person. By subselect or join
- Select phone numbers belonging to each person. By subselect or join
- Loop through the results, building a Lucene index.
- people: ~ 8 million records (could be larger in the future, maybe
- addresses ~8 million records
- contacts: ~10 million records
2x dual core 3GHz intel Xeon
1-2 Gb RAM
It seems that the PostgreSQL driver fetches the whole result, storing
it in memory, causing Java to run out of memory.
We have tried increasing memory for the program (-Xmn100M -Xms512M -
Xmx512M), but we only have one gigabyte of memory to work with.
Does anyone have any suggestions on how to deal with this, other that
throwing more hardware at the problem? Cursors?
Any help is much appreciated!
pgsql-jdbc by date
|Next:||From: Roland Walter||Date: 2006-08-24 08:30:38|
|Subject: Re: Large selects|
|Previous:||From: surabhi.ahuja||Date: 2006-08-24 05:57:30|
|Subject: Re: [JDBC] org.postgresql.util.PSQLException: An I/O error occured while sending to the backend|