Re: speed up verifying UTF-8

On 29/06/2021 14:20, John Naylor wrote:
> I still wasn't quite happy with the churn in the regression tests, so
> for v13 I gave up on using both the existing utf8 table and my new one
> for the "padded input" tests, and instead just copied the NUL byte test
> into the new table. Also added a primary key to make sure the padded
> test won't give weird results if a new entry has a duplicate description.
>
> I came up with "highbit_carry" as a more descriptive variable name than
> "x", but that doesn't matter a whole lot.
>
> It also occurred to me that if we're going to check one 8-byte chunk at
> a time (like v12 does), maybe it's only worth it to load 8 bytes at a
> time. An earlier version did this, but without the recent tweaks. The
> worst-case scenario now might be different from the one with 16-bytes,
> but for now just tested the previous worst case (mixed2).

I tested the new worst case scenario on my laptop:

gcc master:

chinese | mixed | ascii | mixed16 | mixed8
---------+-------+-------+---------+--------
1311 | 758 | 405 | 583 | 725

gcc v13:

chinese | mixed | ascii | mixed16 | mixed8
---------+-------+-------+---------+--------
956 | 472 | 160 | 572 | 939

mixed16 is the same as "mixed2" in the previous rounds, with
'123456789012345ä' as the repeating string, and mixed8 uses '1234567ä',
which I believe is the worst case for patch v13. So v13 is somewhat
slower than master in the worst case.

Hmm, there's one more simple trick we can do: We can have a separate
fast-path version of the loop when there are at least 8 bytes of input
left, skipping all the length checks. With that:

gcc v14:
chinese | mixed | ascii | mixed16 | mixed8
---------+-------+-------+---------+--------
737 | 412 | 94 | 476 | 725

All the above numbers were with gcc 10.2.1. For completeness, with clang
11.0.1-2 I got:

clang master:
chinese | mixed | ascii | mixed16 | mixed8
---------+-------+-------+---------+--------
1044 | 724 | 403 | 930 | 603
(1 row)

clang v13:
chinese | mixed | ascii | mixed16 | mixed8
---------+-------+-------+---------+--------
596 | 445 | 79 | 417 | 715
(1 row)

clang v14:
chinese | mixed | ascii | mixed16 | mixed8
---------+-------+-------+---------+--------
600 | 337 | 93 | 318 | 511

Attached is patch v14 with that optimization. It needs some cleanup, I
just hacked it up quickly for performance testing.

- Heikki