INSERTing lots of data

From: Joachim Worringen <joachim(dot)worringen(at)iathh(dot)de>
To: pgsql-general(at)postgresql(dot)org
Subject: INSERTing lots of data
Date: 2010-05-28 09:41:19
Message-ID: 4BFF8FBF.2050809@iathh.de
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Greetings,

my Python application (http://perfbase.tigris.org) repeatedly needs to
insert lots of data into an exsting, non-empty, potentially large table.
Currently, the bottleneck is with the Python application, so I intend to
multi-thread it. Each thread should work on a part of the input file.

I already multi-threaded the query part of the application, which
requires to use one connection per thread - cursors a serialized via a
single connection.

Provided that
- the threads use their own connection
- the threads perform all INSERTs within a single transaction
- the machine has enough resources

will I get a speedup? Or will table-locking serialize things on the
server side?

Suggestions for alternatives are welcome, but the data must go through
the Python application via INSERTs (no bulk insert, COPY etc. possible)

thanks, Joachim

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Szymon Guz 2010-05-28 09:48:16 Re: INSERTing lots of data
Previous Message Piotr Kublicki 2010-05-28 08:44:10 Re: Download