Re: Speed up COPY FROM text/CSV parsing using SIMD

Hi,

On Tue, 1 Sept 2026 at 07:41, Greg Burd <greg(at)burd(dot)me> wrote:
>
> For anyone following along, the mechanism: commit e0a3a3fd53 added
> CopyReadLineTextSIMDHelper, which refills the input buffer once fewer
> than sizeof(Vector8) bytes remain:
>
> if (copy_buf_len - input_buf_ptr < sizeof(Vector8)) /* reads ahead */
>
> whereas scalar CopyReadLineText only refills when actually empty:
>
> if (input_buf_ptr >= copy_buf_len || need_data)
>
> That difference is the issue. CopyConvertBuf() deliberately tolerates an
> invalid byte sequence after the \. end-of-copy marker (see the comment
> near its convertedlen == 0 block), deferring the error. But
> CopyLoadInputBuf() raises it the moment input_reached_error is set. The
> SIMD read-ahead reaches that raise before the scalar loop has consumed
> \., so a file ending in \.\n<valid bytes><invalid byte> errors out where
> the scalar path would have stopped cleanly at \.

Thank you for great explanation!

On Mon, 31 Aug 2026 at 20:48, Nazir Bilal Yavuz <byavuz81(at)gmail(dot)com> wrote:
>
> That seems like the easiest fix without complicating the code. I will
> review this code in more detail tomorrow, but I think adding the
> reproducer above as a test case makes sense to me.

I re-checked the patch and the fix LGTM. My only comment is adding the
reproducer as a test case.

--
Regards,
Nazir Bilal Yavuz
Microsoft