Skip site navigation (1) Skip section navigation (2)

Re: Import from CSV - Questions

From: "Magnus Hagander" <mha(at)sollentuna(dot)net>
To: "Andreas Pflug" <pgadmin(at)pse-consulting(dot)de>
Cc: "Joel Hainley" <joelh(dot)pg(at)gmail(dot)com>,"Dave Page" <dpage(at)vale-housing(dot)co(dot)uk>,<pgadmin-hackers(at)postgresql(dot)org>
Subject: Re: Import from CSV - Questions
Date: 2006-03-14 21:10:44
Message-ID: 6BCB9D8A16AC4241919521715F4D8BCEA0F86C@algol.sollentuna.se (view raw or flat)
Thread:
Lists: pgadmin-hackers
> > Umm. That would kind of assume it's written down in a 
> comprehensible 
> > way. Which it isn't, of course :-)
> >
> > The general ideas so far have been, off the top of my head:
> >
> > * Pluginnable set of "readers" and "writers". Originally I'd see 
> > postgresql, odbc, xml and possibly csv. Pg driver would be 
> optimised 
> > to use COPY when available.
> >
> > * Pluginnable set of "transforms" that would operate on the 
> rows. By 
> > default things like copy and concatenate and maybe regexp. Future 
> > enhancement would be a python extension, as Dave mentioned. (Or 
> > really, anything else)
> >
> > * I was envisioning a split of say "package", "job", "step" 
> (terms of 
> > course subject to discussion). package basically a set of 
> job, job a 
> > set of steps. Things like connections would be defined at the "job" 
> > level, along wiht parmaeters for transaction control etc. 
> (So you can 
> > use it to transfer 10 different tables within a single transaction, 
> > something I need all the time).
> >
> > * I'd like to see the job format stored as XML with a well defined 
> > schema, so different appliations can generate it - both manually 
> > (GUI-wise from pgadmin and phppgadmin etc) and automatically.
> >
> > * The "engine" should be available both as a commandline 
> tool (which 
> > must not require X libraries etc, because it should be deployable
> > "everywhere") and as  acommand inside pgadmin (like MS DTS)
> >
> >
> > Um. I think that's about it. I had some sketches of classes and 
> > interfaces around (not complete, but an idea), but I can't 
> find them 
> > :(
> >
> >   
> This sounds like an awful lot of work.
> A somewhat reduced version (IIRC Dave and me discussed 
> something like the following briefly) a more raw import 
> (maybe into temp tables) could be a big step, giving the 
> admin the chance to create views on that tables that do the 
> extractions he likes. PostgreSQL already has all functions 
> you'd like, no need to reimplement them.

it is. That's why I wanted to do that first, but I wanted ot have some
sort of generic framework ready first so it could be expanded on later.
I'm definitly not saying it should have all that from the beginning :-)

//Magnus

pgadmin-hackers by date

Next:From: Dave PageDate: 2006-03-17 08:34:04
Subject: Re: [Fwd: Grid code cleanup]
Previous:From: Andreas PflugDate: 2006-03-14 20:54:51
Subject: Re: Import from CSV - Questions

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group