Reducing memory usage of insert into select operations?

From: Francisco Reyes <lists(at)stringsutils(dot)com>
To: PostgreSQL general <pgsql-general(at)postgresql(dot)org>
Subject: Reducing memory usage of insert into select operations?
Date: 2008-07-18 01:21:13
Message-ID: cone.1216344073.438238.29970.1000@zoraida.natserv.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Redhat 4
postgresql 8.3.3
Memory: 12GB

While doing a couple of operations of the type
insert into <table> select from <some other table>

The OS triggered the out of memory killer (oom-killer).

After some research and trial/error I found it was the inserts.
I see one of the inserts is using up 12GB!

How can I reduce the usage?
Postgresql.conf settings.
shared_buffers = 3GB
temp_buffers = 64MB # min 800kB
work_mem = 256MB # min 64kB
maintenance_work_mem = 1GB

Reducing work_mem would help?

The table I am selecting from has a few hundred million rows.
The table I am inserting into has partitions. I am benchmarking breaking up
a large table into smaller partitions.

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Bruce Momjian 2008-07-18 02:22:52 Re: TODO list and "hyphen"
Previous Message Sushant Sinha 2008-07-18 01:16:24 Re: [GENERAL] Fragments in tsearch2 headline