Re: Speed up COPY FROM text/CSV parsing using SIMD

On Wed, Sep 9, 2026, at 6:12 AM, Nazir Bilal Yavuz wrote:
> Hi,
>
> On Tue, 8 Sept 2026 at 23:58, Nathan Bossart <nathandbossart(at)gmail(dot)com> wrote:
>>
>> On Tue, Sep 08, 2026 at 11:58:42AM -0400, Greg Burd wrote:
>> > The test writes one full input buffer (INPUT_BUF_SIZE, 65536 bytes) down a
>> > COPY FROM PROGRAM pipe: a 65524-byte line, then a complete \. marker, then
>> > 8 trailing bytes. That leaves 11 bytes unexamined when the SIMD path wants
>> > to refill, which is under sizeof(Vector8) and already holds the whole
>> > marker. The writer then dribbles a byte every 50ms and keeps the pipe open,
>> > so a speculative load blocks rather than seeing EOF. statement_timeout
>> > bounds the failure, so a regression fails the test instead of hanging the
>> > suite. Skipped on Windows, where the blocking pipe and SIGPIPE behavior
>> > isn't reliable.
>>
>> I'm not sure this particular corner case deserves its own TAP test.
>
> I have similar feelings.

It was just a first swing at a test, and I agree with both of you it felt
heavyweight for the issue at hand. But it does capture the issue.

>> IMHO
>> that is disproportionately expensive. It really only serves to prevent us
>> from removing the backslash scan from the SIMD path, which I don't think
>> we're likely to do.
>
> I too thought it would be expensive but it wasn't on my machine; it
> took 0.5s, which is lower than I expected.
>
>
>> Furthermore, I'm not aware of any existing examples
>> where a test writes and then invokes its own Perl script, and I can't get
>> too excited about being the first to commit something like that. So, I'd
>> prefer to leave this test out for now. If the new test fit cleanly into an
>> existing suite or was less magical, I might be more eager to include it,
>> but it's probably still unlikely to be worth the energy and maintenance
>> risk.

Yeah, that's a solid judgement call. One I should have made myself.

> Perhaps we can save this Perl script as another file and invoke that
> file. This seems better to me. However, I agree that adding this test
> might be overkill given the current circumstances.

It's overkill, let's focus and get the fix in.

> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft

best.

-greg