[BUG] hstore integer overflow when constructing large values

Hi,

My colleague Man Zeng reported an issue where hstore could fail with a
very large and unexpected memory allocation request. I investigated the
issue and found an integer overflow when constructing an hstore from
multiple large values.

For example:

CREATE EXTENSION hstore;
CREATE TABLE t(a text, b text, c text);

INSERT INTO t VALUES (repeat('x', 720000000), NULL, NULL);
UPDATE t SET b = repeat('y', 720000000);
UPDATE t SET c = repeat('z', 720000000);

SELECT hstore(t) FROM t;

This produces:
ERROR: invalid memory alloc request size 18446744071574584355

The problem is in `hstoreUniquePairs()`. `buflen` is an `int32`, and the
total length of all keys and values is accumulated into it without an
overflow check:

*buflen += res->keylen + ((res->isnull) ? 0 : res->vallen);

Each individual value in the example is within the hstore value length
limit, but their combined length is not. In this case, the accumulated
length is:
(720000000 + 1) * 3 = 2160000003

which exceeds `INT_MAX`. `buflen` therefore overflows before
`hstorePairs()` calculates the size of the resulting hstore. The negative
value eventually gets converted to `Size` when passed to `palloc()`,
resulting in the very large allocation size shown in the error message.

The attached patch checks the accumulated length before updating `buflen`,
so that an oversized hstore is rejected before the `int32` overflow can
occur. It also checks the complete size calculated by `CALCDATASIZE()`,
since the HEntry array and hstore header must fit in the resulting datum as
well.

I kept the existing `int32` arguments of `hstoreUniquePairs()` and
`hstorePairs()` unchanged rather than changing the exported interfaces.

The patch reports the size limit error consistently with the existing
hstore length checks.

I have not added the reproducer to the regression tests because it requires
constructing more than 2GB of input data. I couldn't find a reasonably
small SQL-level test case that exercises the same overflow.

Thanks to Man Zeng for reporting the issue.

Patch attached. Comments are welcome.

--
Thanks,
Tender Wang