Re: Fuzz testing COPY FROM parsing

On 05/02/2021 21:16, Andrew Dunstan wrote:
>
> On 2/5/21 10:54 AM, Stephen Frost wrote:
>> * Heikki Linnakangas (hlinnaka(at)iki(dot)fi) wrote:
>>> I ran it for about 2 h on my laptop with the patch I was working on [2]. It
>>> didn't find any crashes, but it generated about 1300 input files that it
>>> considered "interesting" based on code coverage analysis. When I took those
>>> generated inputs, and ran them against unpatched and patched server, some
>>> inputs produced different results. So that revealed a couple of bugs in the
>>> patch. (I'll post a fixed patched version on that thread soon.)
>>>
>>> I hope others find this useful, too.
>> Nice! I wonder if there's a way to have a buildfarm member or other
>> system doing this automatically on new commits and perhaps adding
>> coverage for other things like the JSON code..
>
> Not easily in the buildfarm as it is today. We can easily create modules
> for extensions and other things that don't require modification of core
> code, but things that require patching core code are a whole different
> story.

It might be possible to call the fuzzer's HF_ITER() function from a C
extension instead. So you would run a query like "SELECT
next_fuzz_iter()" in a loop, and next_fuzz_iter() would be a C function
that calls HF_ITER(), and executes the actual query with SPI.

That said, I don't think it's important to run the fuzzer in the
buildfarm. It should be enough to do that every once in a while, when
you modify the COPY FROM code (or something else that you want to fuzz
test). But we could easily include the test inputs generated by the
fuzzer in the regular tests. We've usually been very frugal in adding
tests, though, to keep the time it takes to run all the tests short.

- Heikki