From: | John Naylor <john(dot)naylor(at)enterprisedb(dot)com> |
---|---|
To: | Heikki Linnakangas <hlinnaka(at)iki(dot)fi> |
Cc: | pgsql-hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: Perform COPY FROM encoding conversions in larger chunks |
Date: | 2021-02-02 21:42:31 |
Message-ID: | CAFBsxsESOT341t-CTNTgSe+U2xVW_TnT2g=Aq20tb9VPCbtk3A@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Mon, Feb 1, 2021 at 12:15 PM Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:
> Thanks. I fixed it slightly differently, and also changed LocalToUtf()
> to follow the same pattern, even though LocalToUtf() did not have the
> same bug.
Looks good to me.
> I added a bunch of tests for various built-in conversions.
Nice! I would like to have utf8 tests for every category of invalid byte
(overlong, surrogate, 5 bytes, etc), but it's not necessary for this patch.
> I spent some time refactoring and adding comments all around the patch,
> hopefully making it all more clear. One notable difference is that I
> renamed 'raw_buf' (which exists in master too) to 'input_buf', and
> renamed 'conversion_buf' to 'raw_buf'. I'm going to read through this
> patch again another day with fresh eyes, and also try to add some tests
> for the corner cases at buffer boundaries.
The comments and renaming are really helpful in understanding that file!
Although a new patch is likely forthcoming, I did take a brief look and
found the following:
In copyfromparse.c, this is now out of date:
* Read the next input line and stash it in line_buf, with conversion to
* server encoding.
One of your FIXME comments seems to allude to this, but if we really need a
difference here, maybe it should be explained:
+#define INPUT_BUF_SIZE 65536 /* we palloc INPUT_BUF_SIZE+1 bytes */
+#define RAW_BUF_SIZE 65536 /* allocated size of the buffer */
Lastly, it looks like pg_do_encoding_conversion_buf() ended up in 0003
accidentally?
--
John Naylor
EDB: http://www.enterprisedb.com
From | Date | Subject | |
---|---|---|---|
Next Message | Robert Haas | 2021-02-02 22:11:48 | Re: Background writer and checkpointer in crash recovery |
Previous Message | Tom Lane | 2021-02-02 21:17:49 | Re: [PATCH] Doc: improve documentation of oid columns that can be zero. (correct version) |