Skip site navigation (1) Skip section navigation (2)

Re: Optimizing COPY

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Optimizing COPY
Date: 2008-10-30 13:29:33
Message-ID: (view raw, whole thread or download thread mbox)
Lists: pgsql-hackers
Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:
> The basic idea is to replace the custom loop in CopyReadLineText with 
> memchr(), because memchr() is very fast. To make that possible, perform 
> the client-server encoding conversion on each raw block that we read in, 
> before splitting it into lines. That way CopyReadLineText only needs to 
> deal with server encodings, which is required for the memchr() to be safe.

Okay, so of course the trick with that is the block boundary handling.
The protocol says the client can break the data apart however it likes.
I see you've tried to deal with that, but this part seems wrong:

> ! 			if (convertable_bytes == 0)
> ! 			{
> ! 				/*
> ! 				 * EOF, and there was some unconvertable chars at the end.
> ! 				 * Call pg_client_to_server on the remaining bytes, to
> ! 				 * let it throw an error.
> ! 				 */
> ! 				cvt = pg_client_to_server(raw, inbytes);
> ! 				Assert(false); /* pg_client_to_server should've errored */
> ! 			}

You're not (AFAICS) definitely at EOF here; you might just have gotten
a pathologically short message.

			regards, tom lane

In response to

pgsql-hackers by date

Next:From: Heikki LinnakangasDate: 2008-10-30 13:31:14
Subject: Re: User defined I/O conversion casts
Previous:From: Tom LaneDate: 2008-10-30 13:21:46
Subject: Re: Hot Standby: Caches and Locks

Privacy Policy | About PostgreSQL
Copyright © 1996-2017 The PostgreSQL Global Development Group