| From: | shihao zhong <zhong950419(at)gmail(dot)com> |
|---|---|
| To: | pgsql-hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org> |
| Cc: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
| Subject: | Reject unassigned bytes in single-byte encodings |
| Date: | 2026-10-26 04:57:45 |
| Message-ID: | CAGRkXqTdQ5Zin7Ayw-QY2pjnWvVOrm8x6+0ruSrKsT2FguRBaQ@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi,
In [1] Tom said the real fix for the psql problem was the WIN1252
validator. Here is a patch for master.
Thirteen single-byte encodings have bytes with no character. I took
the lists from each *_to_utf8.map:
ISO_8859_6 ISO_8859_7 ISO_8859_8 LATIN3 WIN874
WIN1250 WIN1251 WIN1252 WIN1253 WIN1254 WIN1255
WIN1257 WIN1258
For WIN1252 they are 0x81 0x8D 0x8F 0x90 0x9D. Today these bytes are
accepted, then fail to convert to any other encoding. Bug #14197 [2]
is one report of that.
The patch rejects them in the verifystr functions. It also closes two
other ways in. chr(129) skips the validator, so it now checks the
result. And the LATIN2 to WIN1250 table mapped five bytes into WIN1250
holes, so those now fail to convert.
A database that already has such bytes can still be dumped, but the
restore fails on COPY for that table. pg_upgrade keeps the bytes as
they are. Maybe pg_upgrade should check for them as well////
*The extra check costs something.* COPY of 1M rows of 60 bytes:
WIN1252 ASCII only 0.221 s -> 0.226 s (noise)
WIN1252 50% non-ASCII 0.218 s -> 0.261 s (+20%)
ISO 8859-6 100% non-ASCII 0.219 s -> 0.271 s (+23%)
UTF8 databases do not pay for it.
Another idea is to map the holes to characters instead, for example
0x81 to U+0081. That costs nothing and keeps old data readable. But
our maps would no longer match the Unicode.org files.
^^^ only works cleanly for holes in 0x80-0x9F, which covers 5/13.
Thanks,
Shihao
| Attachment | Content-Type | Size |
|---|---|---|
| v1-0002-Add-tests-for-unassigned-bytes-in-single-byte-enc.patch | application/octet-stream | 5.3 KB |
| v1-0001-Reject-unassigned-bytes-in-single-byte-encodings.patch | application/octet-stream | 13.7 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | shveta malik | 2026-10-26 04:58:58 | Re: Proposal: Conflict log history table for Logical Replication |
| Previous Message | Henson Choi | 2026-10-26 04:46:13 | Re: [DEPARSE] AT TIME ZONE and similar as a function in FROM do not read back |