spg_quad_inner_consistent() doesn't check all four box corners

In the thread for Bug #19597, I wrote:

> I may have found 3 other possible bugs involving SP-Gist in the course
> of investigating this, so I will share those in separate threads when
> I return to this topic.

Here's the first of these. All three were found with fuzzing. This
wrong-result repro piggybacks on the regression test from the above
bug fix:

create table spgist_quad_fp_tbl(p point);
insert into spgist_quad_fp_tbl select point(i, 5) from
generate_series(1, 300) i;
create index spgist_quad_fp_idx on spgist_quad_fp_tbl using spgist(p);

insert into spgist_quad_fp_tbl values (point(3, 5 + 1e-6));
insert into spgist_quad_fp_tbl values (point(114 - 5e-7, 5 + 1e-6));
insert into spgist_quad_fp_tbl values (point(200, 5 - 5e-6));

set enable_seqscan = off;

select count(*) from spgist_quad_fp_tbl
where p <@ box(point(0, 4.99999), point(400, 4.9999995));
count
-------
0
(1 row)

reset enable_seqscan;
set enable_indexscan = off;

select count(*) from spgist_quad_fp_tbl
where p <@ box(point(0, 4.99999), point(400, 4.9999995));
count
-------
1
(1 row)

reset all;
--

spg_quad_inner_consistent() prunes quadrants that don't contain a
corner of the box. If the box lies well below the centroid and
straddles the y axis, it finds we need to visit quadrants 2 and 3. So
far so good. In the above case, the top edge of the box is within
EPSILON of the centroid. The top right corner goes to quadrant 1, and
the left corners go to quadrant 3. The bottom right corner belongs in
quadrant 2, but the code doesn't actually test that, it tests the
upper left corner again, so it visits quadrants 1 and 3 and skips the
quadrant we need which is 2. The fix is simple:

p = boxQuery->low;
r |= 1 << getQuadrant(centroid, &p);
p.y = boxQuery->high.y;
r |= 1 << getQuadrant(centroid, &p);
p = boxQuery->high;
r |= 1 << getQuadrant(centroid, &p);
- p.x = boxQuery->low.x;
+ p.y = boxQuery->low.y;
r |= 1 << getQuadrant(centroid, &p);

The attached does this and adds a regression test. I'll commit this soon.

--
John Naylor
Amazon Web Services