Re: [sqlsmith] Failed assertion in brin_minmax_multi_distance_float4 on REL_14_STABLE

Hi,

On 11/4/21 17:53, Justin Pryzby wrote:
> On Thu, Nov 04, 2021 at 09:46:49AM +0100, Andreas Seltenreich wrote:
>> sqlsmith triggers the following assertion when testing REL_14_STABLE:
>>
>> TRAP: FailedAssertion("a1 <= a2", File: "brin_minmax_multi.c", Line: 1879, PID: 631814)
>>
>> I can reproduce it with the following query on a fresh regression
>> database:
>>
>> insert into public.brintest_multi (float4col) values (real 'nan');
>>
>> The branch was at f6162c020c while testing, backtrace below.
>
> I couldn't reproduce this, but it reminds me of this one, which we also had
> trouble reproducing.
>

I can reproduce that just fine - all I had to do was 'make installcheck'
and then connect to the regression db and run the insert.

It seems to be a simple case of confusion in handling of NaN values. We
do sort them correctly (by calling float4_lt), but given two values

arg1 = nan (0x400000)
arg2 = 0.0909090936

then a simple comparison does not give the expected result

(arg1 < arg2)
(arg1 == arg2)
(arg1 > arg2)

all evaluate to false, which is why the assert fails. So I guess the
distance function for float4 (and probably float8 too) need a couple
more lines checking NaN.

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company