| From: | Chengpeng Yan <chengpeng_yan(at)outlook(dot)com> |
|---|---|
| To: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
| Cc: | PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, Jeff Davis <pgsql(at)j-davis(dot)com> |
| Subject: | Re: [PATCH] Fix encoding-dependent LIKE/regex selectivity estimates |
| Date: | 2026-11-19 10:30:37 |
| Message-ID: | 7F684DCD-4105-44A9-B764-6F3377D75711@outlook.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi,
> On Nov 19, 2026, at 09:25, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> I wrote:
>> I'm not sure what I think about this proposal. I agree it looks dumb
>> to estimate the selectivity of "é" differently from "e", and even more
>> so for it to depend on the encoding. However, I believe this per-byte
>> counting was intentional, with an eye to Far Eastern character sets.
>> A single ideogram of Chinese or Japanese probably carries about the
>> same selectivity as a word of English, so giving it the same
>> selectivity as "e" likewise doesn't sound great. I'm not trying to
>> claim that per-byte counting is ideal, but there is more here than
>> meets the eye, so I'm loath to change a longstanding behavior on
>> the basis of no discussion or evidence. How can we arrive at a
>> behavior that is sensible for both of those cases?
>
> After noodling about that for awhile, I have a potential answer.
> Basically we need (I think) to distinguish letter-y characters
> from ideogram-y characters. But there is a serviceable, and
> fast, approximation to that in our infrastructure already:
> check pg_encoding_dsplen() and see if it's 1 or 2. Double-width
> characters can be counted on to carry more information than
> regular-width ones. I don't have an immediate proposal about
> what selectivity multiplier to assign to double-width characters,
> but this approach seems like there's potential to accomplish
> what the original hack hoped for without the encoding dependency.
>
> regards, tom lane
Agreed, and v2 takes that approach. dsplen describes the character
rather than its storage: "é" is narrow in both LATIN1 and UTF-8, and a
double-width character gets extra weight, so an ideograph is estimated
to match far fewer rows than a letter. What the estimate depends on is
how wide the character is, not how many bytes the encoding needs.
The multiplier is the part I can't justify rigorously: how much a
double-width character narrows a match probably differs between scripts,
and I saw no way to derive one value from data. I picked 3 because wide
characters come mostly from the CJK scripts and their common ones take
three bytes in UTF-8, so the estimates in the usual case stay as they
are. Happy to change it.
Your other comments are all in v2. Thanks for the review and the dsplen
suggestion.
--
Best regards,
Chengpeng Yan
| Attachment | Content-Type | Size |
|---|---|---|
| v2-0001-Charge-pattern-selectivity-per-character-not-per-.patch | application/octet-stream | 6.6 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Amit Kapila | 2026-11-19 10:31:25 | Re: Add a hook for handling logical decoding messages on subscribers. |
| Previous Message | vignesh C | 2026-11-19 10:06:34 | Re: Proposal: Conflict log history table for Logical Replication |