Re: Speed up COPY FROM text/CSV parsing using SIMD

Hi,

On Tue, 1 Sept 2026 at 22:24, Nathan Bossart <nathandbossart(at)gmail(dot)com> wrote:
>
> On Tue, Sep 01, 2026 at 11:17:03AM +0300, Nazir Bilal Yavuz wrote:
> > I re-checked the patch and the fix LGTM. My only comment is adding the
> > reproducer as a test case.
>
> Committed, thanks for looking. I'm concerned about portability and
> encoding issues with such a test, so I didn't add it yet. But I'm not
> opposed to adding a test along these lines as a separate effort.

After the problem you found, I had another LLM review the code and it
found one issue:

When fewer than one SIMD vector of bytes (16 in this case) remain
buffered, CopyReadLineTextSIMDHelper() refills the input before
falling back to scalar parsing. If those buffered bytes already
contain a complete '\.' (end-of-copy marker) and the source is an open
FIFO with no more data, the refill blocks, so COPY fails to recognize
the marker until more data arrives or the writer closes the FIFO.
Reproducer SQL script is attached, you can see that this causes a
hang.

One potential fix is checking for an end-of-copy marker when fewer
than one SIMD vector of bytes remains in the buffer and we decide to
load more data into it. I haven't benchmarked this solution yet but it
could potentially cause a slowdown when we load data in smaller
chunks. Otherwise, I don't think this solution will cause a slowdown.
I am planning to work on this tomorrow.

Any opinions on the bug or the potential solution?

--
Regards,
Nazir Bilal Yavuz
Microsoft