Re: Import from CSV - Questions

Magnus Hagander wrote:
>> Magnus, any chance of getting a look at what you currently
>> have to help guide me in the appropriate direction?
>>
>
> Umm. That would kind of assume it's written down in a comprehensible
> way. Which it isn't, of course :-)
>
> The general ideas so far have been, off the top of my head:
>
> * Pluginnable set of "readers" and "writers". Originally I'd see
> postgresql, odbc, xml and possibly csv. Pg driver would be optimised to
> use COPY when available.
>
> * Pluginnable set of "transforms" that would operate on the rows. By
> default things like copy and concatenate and maybe regexp. Future
> enhancement would be a python extension, as Dave mentioned. (Or really,
> anything else)
>
> * I was envisioning a split of say "package", "job", "step" (terms of
> course subject to discussion). package basically a set of job, job a set
> of steps. Things like connections would be defined at the "job" level,
> along wiht parmaeters for transaction control etc. (So you can use it to
> transfer 10 different tables within a single transaction, something I
> need all the time).
>
> * I'd like to see the job format stored as XML with a well defined
> schema, so different appliations can generate it - both manually
> (GUI-wise from pgadmin and phppgadmin etc) and automatically.
>
> * The "engine" should be available both as a commandline tool (which
> must not require X libraries etc, because it should be deployable
> "everywhere") and as acommand inside pgadmin (like MS DTS)
>
>
> Um. I think that's about it. I had some sketches of classes and
> interfaces around (not complete, but an idea), but I can't find them :(
>
>
This sounds like an awful lot of work.
A somewhat reduced version (IIRC Dave and me discussed something like
the following briefly) a more raw import (maybe into temp tables) could
be a big step, giving the admin the chance to create views on that
tables that do the extractions he likes. PostgreSQL already has all
functions you'd like, no need to reimplement them.

Regards,
Andreas