Re: Optimizing COPY

Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:
> The basic idea is to replace the custom loop in CopyReadLineText with
> memchr(), because memchr() is very fast. To make that possible, perform
> the client-server encoding conversion on each raw block that we read in,
> before splitting it into lines. That way CopyReadLineText only needs to
> deal with server encodings, which is required for the memchr() to be safe.

Okay, so of course the trick with that is the block boundary handling.
The protocol says the client can break the data apart however it likes.
I see you've tried to deal with that, but this part seems wrong:

> ! if (convertable_bytes == 0)
> ! {
> ! /*
> ! * EOF, and there was some unconvertable chars at the end.
> ! * Call pg_client_to_server on the remaining bytes, to
> ! * let it throw an error.
> ! */
> ! cvt = pg_client_to_server(raw, inbytes);
> ! Assert(false); /* pg_client_to_server should've errored */
> ! }

You're not (AFAICS) definitely at EOF here; you might just have gotten
a pathologically short message.

regards, tom lane