Re: [HACKERS] Inefficiencies in COPY command

From: Bruce Momjian <maillist(at)candle(dot)pha(dot)pa(dot)us>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Wayne Piekarski <wayne(at)senet(dot)com(dot)au>, pgsql-hackers(at)postgreSQL(dot)org
Subject: Re: [HACKERS] Inefficiencies in COPY command
Date: 1999-09-27 18:04:42
Message-ID: 199909271804.OAA10131@candle.pha.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Yes, I too would be interested in any code that would speed up COPY
without loosing modularity or portability.

Please let us know if you get a patch we can include in our source tree.

> Wayne Piekarski <wayne(at)senet(dot)com(dot)au> writes:
> > So I made some changes to CopyAttributeOut so that it escapes the string
> > initially into a temporary buffer (allocated onto the stack) and then
> > feeds the whole string to the CopySendData which is a lot more efficient
> > because it can blast the whole string in one go, saving about 1/3 to 1/4
> > the number of memcpy and so on.
>
> copy.c is pretty much of a hack job to start with, IMHO. If you can
> speed it up without making it even uglier, have at it! However, it
> also has to be portable, and what you've done here:
>
> > CopyAttributeOut(FILE *fp, char *server_string, char *delim, int is_array)
> > {
> > char *string;
> > char c;
> > + char __buffer [(strlen (server_string) * 2) + 4]; /* Use +4 for safety */
>
> is not portable --- variable-sized local arrays are a gcc-ism. (I use
> 'em a lot too, but not in code intended for public release...) Also,
> be careful that you don't introduce any assumptions about maximum
> field or tuple width; we want to get rid of those, not add more.
>
> > to also make changes to remove all use of sprintf when numbers
> > and floats are being converted into strings.
> ^^^^^^^^^^
>
> While formatting an int is pretty simple, formatting a float is not so
> simple. I'd be leery of replacing sprintf with quick-hack float
> conversion code. OTOH, if someone wanted to go to the trouble of doing
> it *right*, using our own code would tend to yield more consistent
> results across different OSes, which would be a Good Thing. I'm not
> sure it'd be any faster than the typical sprintf, but it might be worth
> doing anyway.
>
> (My idea of *right* is the guaranteed-inverse float<=>ASCII routines
> published a few years ago in some SIGPLAN proceedings ... I've got the
> proceedings, and I even know approximately where they are, but I don't
> feel like excavating for them right now...)
>
> regards, tom lane
>
>

--
Bruce Momjian | http://www.op.net/~candle
maillist(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Bruce Momjian 1999-09-27 18:05:29 Re: [HACKERS] another DECIMAL problem
Previous Message Oleg Bartunov 1999-09-27 17:59:50 NULL as an argument in plpgsql functions