Re: Speed up COPY FROM text/CSV parsing using SIMD

Hi,

On Tue, 8 Sept 2026 at 23:58, Nathan Bossart <nathandbossart(at)gmail(dot)com> wrote:
>
> On Tue, Sep 08, 2026 at 11:58:42AM -0400, Greg Burd wrote:
> > The test writes one full input buffer (INPUT_BUF_SIZE, 65536 bytes) down a
> > COPY FROM PROGRAM pipe: a 65524-byte line, then a complete \. marker, then
> > 8 trailing bytes. That leaves 11 bytes unexamined when the SIMD path wants
> > to refill, which is under sizeof(Vector8) and already holds the whole
> > marker. The writer then dribbles a byte every 50ms and keeps the pipe open,
> > so a speculative load blocks rather than seeing EOF. statement_timeout
> > bounds the failure, so a regression fails the test instead of hanging the
> > suite. Skipped on Windows, where the blocking pipe and SIGPIPE behavior
> > isn't reliable.
>
> I'm not sure this particular corner case deserves its own TAP test.

I have similar feelings.

> IMHO
> that is disproportionately expensive. It really only serves to prevent us
> from removing the backslash scan from the SIMD path, which I don't think
> we're likely to do.

I too thought it would be expensive but it wasn't on my machine; it
took 0.5s, which is lower than I expected.

> Furthermore, I'm not aware of any existing examples
> where a test writes and then invokes its own Perl script, and I can't get
> too excited about being the first to commit something like that. So, I'd
> prefer to leave this test out for now. If the new test fit cleanly into an
> existing suite or was less magical, I might be more eager to include it,
> but it's probably still unlikely to be worth the energy and maintenance
> risk.

Perhaps we can save this Perl script as another file and invoke that
file. This seems better to me. However, I agree that adding this test
might be overkill given the current circumstances.

--
Regards,
Nazir Bilal Yavuz
Microsoft