Skip site navigation (1) Skip section navigation (2)

Re: import function's progress report

From: Magnus Hagander <magnus(at)hagander(dot)net>
To: Dave Page <dpage(at)pgadmin(dot)org>
Cc: Quan Zongliang <quanzongliang(at)gmail(dot)com>, pgadmin-hackers(at)postgresql(dot)org
Subject: Re: import function's progress report
Date: 2008-12-31 10:58:12
Message-ID: 495B5044.8020905@hagander.net (view raw or flat)
Thread:
Lists: pgadmin-hackers
Dave Page wrote:
> Hi
> 
> On Wed, Dec 31, 2008 at 7:14 AM, Quan Zongliang <quanzongliang(at)gmail(dot)com> wrote:
>> Hi, all

>> 3. Columns page:
>> Set data format ( date & number), space trim and column's order
>> I plan to support express, user can input a pgsql express to treat data.
> 
> Hmm, pgsql expressions could be slow as they can only be evaluated in
> insert statements, which is really not good for such a tool. I would
> suggest:
> 
> - Consider embedding Python (which is on the TODO anyway) and allowing
> transforms to be scripted on the client.
> - Convert the import data to a COPY compatible format so it can be fed
> directly to the server without using INSERT which is *much* slower.

Agreed, and a feature I would very much like to see.

FYI, I recently wrote a small tool that did *just* this step (the
conversion while loading). When running both the client and the server
on the same machine, it still came out more than 3x faster if I did the
simple transforms in Python and maintained the COPY format rather than
switching to INSERT format.

Do you plan to support loading multiple tables in one batch
(transaction)? If not initially, it would be great if you could plan for
the ability to do so on in the future - that is, don't make assumptions
that we will not want it. Because that would be a great feature.



//Magnus


In response to

Responses

pgadmin-hackers by date

Next:From: Guillaume LelargeDate: 2008-12-31 11:00:51
Subject: Re: Patch to handle connection limit for roles
Previous:From: Dave PageDate: 2008-12-31 10:24:56
Subject: Re: import function's progress report

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group