Re: [HACKERS] COPY problems with psql / libpq

On 2000-01-20, Tom Lane mentioned:

> "Oliver Elphick" <olly(at)lfix(dot)co(dot)uk> writes:
> > I found that if I broke the first 1000 records into 2 equal parts, all
> > of them were added correctly without error; so I conclude that data
> > is being buffered and lost somewhere in psql or libpq, and the problem is
> > dependent on the amount of data being copied.

The buffering is line-based though and the default buffer is 8192. If
your line is longer than that you're in all sorts of other troubles.

>
> I have the following note in my (much too long) to-do list:
>
> : psql.c doesn't appear to cope correctly with quoted newlines in COPY data;

What's a quoted newline?
a) "<newline>"
b) "\n"
c) \<newline>

Earlier you also mentioned to me something in general about control
characters messing up COPY. Could you give me some details on that so I
can look into it?

> : if one falls just after a buffer boundary, trouble!
> : Does fe-exec.c work either??
>
> (This note is some months old, and may or may not still apply since
> Peter's rework of psql.) It could be that your dataset is hitting this

I haven't touched that code.

> problem or a similar one. A buffer-boundary problem would explain why
> the error seems to be so dataset-specific.

At first I thunk a PQExpBuffer based solution would be the answer, but
as I said above, if you overflow the buffer, you're in trouble anyway.

>
> > copy address from stdin;
> > -- 1000 records written
> > select count(*) from address;
> > PQexec: you gotta get out of a COPY state yourself.
>
> It sure sounds like psql is failing to recognize the trailing \.
> of the COPY data.

The last call in the function psql/copy.c:handleCopyIn is

return !PQendcopy(conn);

and there is no way it can exit earlier. Also the connection seems to be
good, since that's checked right after it returns. The calls to PQputvalue
are not checked for return values, so problems might get missed there, but
that would in any case still point to a problem elsewhere. Gotta pass the
buck to libpq ...

--
Peter Eisentraut Sernanders väg 10:115
peter_e(at)gmx(dot)net 75262 Uppsala
http://yi.org/peter-e/ Sweden