Re: Fw:Re: Fw: gbt_var_consistent in contrib/btree_gist/btree_utils_var.c has internal-node type confusion on the <> strategy, bypassing exclusion constraints

Hi,

On Tue, 16 Jun 2026 at 16:59, 王跃林 <violin0613(at)tju(dot)edu(dot)cn> wrote:

> 0800, 王跃林 wrote:
>
> > gbt_var_node_truncate (btree_utils_var.c:214) truncates internal node keys to a common-prefix length. The resulting bytea can have VARSIZE anywhere from 4 upward. When the truncated VARSIZE is below 8 and that key reaches bit_cmp via the buggy BtreeGistNotEqual branch, bytelen becomes negative. Passed to memcmp as size_t, that is several GB. ASan catches it as negative-size-param. A production build without ASan will eventually SEGV when the read crosses an unmapped page.
>
> Thanks for the report!

In gbt_var_consistent() (contrib/btree_gist/btree_utils_var.c), the <>
(BtreeGistNotEqual) branch is the only strategy that does not distinguish
leaf from internal pages. It applies the type's equality function directly
to the stored key on both bounds:

retval = !(tinfo->f_eq(query, key->lower, ...) &&
tinfo->f_eq(query, key->upper, ...));

That seems to be the problem: on internal pages those keys are
truncated (and for bit/varbit the length header is gone too), so
they aren't really valid values to hand to the type's equality
function. Hence the negative length reaching memcmp().

Would it make sense to just have <> follow the same leaf-vs-internal
pattern as the other strategies; negate equality on a leaf, and
otherwise recurse? An internal page can always contain some
non-equal value, so I don't think we lose anything by descending.
I put together a small patch along those lines (plus a bit(8) test
that goes through internal pages); it's attached if it's useful.
(PS. The test exercises the path but passes on a plain build; the OOB
read shows up under ASan (negative-size-param).)

I didn't touch the fixed-size path in btree_utils_num.c, since its
internal keys look like they're exact lower/upper values, but
please correct me if I'm missing something there.

Regards,
Ayush