Re: Ragged CSV import

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> you mean some sort of filter mechanism?
>
>> COPY .... FILTER function_name ( args) ... ?
>
>> That might work. Then we could provide a couple builtin and people could
>> write others in C or PL/dujour.
>
> Yeah, that's pretty much what I was thinking, although exactly where the
> filter happens and what its arguments/results are is open to debate.

Last time we opened this can of worms here's what it lead us to:

http://archives.postgresql.org/pgsql-hackers/2008-04/msg00222.php

That is, COPY support in the FROM clause. Now the example gets rewritten
as:

SELECT *
FROM function_name(a, b, c)
LATERAL (COPY ... FROM ...);

Or with some more nesting if we don't have LATERAL by the time we get
COPY support in the FROM clause :)

That allows to simplify some things: COPY in FROM always returns text
datums, the function_name can be any callable function (pl/whatever),
and we could import easily data whose representation has to be adapted
to PostgreSQL data type input function (such as timestamps etc)

INSERT INTO t SELECT myparse(a) FROM (COPY ...);

As already said by Andrew on IRC, though, this goes nowhere to help him
address his direct concern, COPY would still choke on input lines that
are containing too few or too many columns...

Regards,
--
dim