Skip site navigation (1) Skip section navigation (2)

Re: Massive Inserts Strategies

From: ashah <ashah(at)opuspro(dot)com>
To: pgsql-performance(at)postgresql(dot)org
Subject: Re: Massive Inserts Strategies
Date: 2006-03-28 15:59:49
Message-ID: 200603281059.49790.ashah@opuspro.com (view raw or flat)
Thread:
Lists: pgsql-performance
I tried this solution, but ran into following problem.

The temp_table has columns (col1, col2, col3).

The original_table has columns (col0, col1, col2, col3)

Now the extra col0 on the original_table is the unique generated ID by the 
database.

How can I make your suggestions work in that case .. ?

On Wednesday 22 March 2006 11:09 am, Jim C. Nasby wrote:
> Load the files into a temp table and go from there...
>
> COPY ... FROM file;
> UPDATE existing_table SET ... WHERE ...;
> INSERT INTO existing_table SELECT * FROM temp_table WHERE NOT EXISTS(
> SELECT * FROM existing_table WHERE ...)
>
> On Wed, Mar 22, 2006 at 10:32:10AM -0500, ashah wrote:
> > I have a database with foreign keys enabled on the schema. I receive
> > different files, some of them are huge. And I need to load these files in
> > the database every night. There are several scenerios that I want to
> > design an optimal solution for -
> >
> > 1. One of the file has around 80K records and I have to delete everything
> > from the table and load this file. The provider never provides a "delta
> > file" so I dont have a way to identify which records are already present
> > and which are new. If I dont delete everything and insert fresh, I have
> > to make around 80K selects to decide if the records exist or not. Now
> > there are lot of tables that have foreign keys linked with this table so
> > unless I disable the foreign keys, I cannot really delete anything from
> > this table. What would be a good practise here?
> >
> > 2. Another file that I receive has around 150K records that I need to
> > load in the database. Now one of the fields is logically a "foreign key"
> > to another table, and it is linked to the parent table via a database
> > generated unique ID instead of the actual value. But the file comes with
> > the actual value. So once again, I have to either drop the foreign key,
> > or make 150K selects to determine the serial ID so that the foreign key
> > is satisfied. What would be a good strategy in this scenerio ?
> >
> > Please pardon my inexperience with database !
> >
> > Thanks,
> > Amit
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 4: Have you searched our list archives?
> >
> >                http://archives.postgresql.org

In response to

Responses

pgsql-performance by date

Next:From: Dave DutcherDate: 2006-03-28 16:18:25
Subject: Re: simple join uses indexes, very slow
Previous:From: MarcosDate: 2006-03-28 15:31:42
Subject: Decide between Postgresql and Mysql (help of comunity)

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group