pgsql: Reintroduce hash_mem cap on buckets in BuildTupleHashTable

Reintroduce hash_mem cap on buckets in BuildTupleHashTable

8f29467c5 removed the hash_mem cap from the nbuckets calculation in
BuildTupleHashTable() and put the trust into the planner to provide that
function with a sane value, which it would only cap at 2^32. There are
various places in the planner which cannot provide good estimates here
and it's been demonstrated that it's not difficult to craft a query
which requests 2^32 buckets.

Here we reintroduce the sanity checks so we don't end up making the hash
table's initial size to something that would exceed hash_mem. This may
cause the hash table to have to grow several times before it fills, but
that seems much better than having the executor OOM due to the request
being too large. Users are also used to work_mem and
hash_mem_multiplier having an effect on query performance, so this
shouldn't be too surprising. It also restores the limits that were
imposed pre-8f29467c5, which were tried and tested.

Bug: #19759
Reported-by: Alperen Keles <alpkeles99(at)gmail(dot)com>
Author: David Rowley <dgrowleyml(at)gmail(dot)com>
Reviewed-by: shihao zhong <zhong950419(at)gmail(dot)com>
Discussion: https://postgr.es/m/19759-acd3f33402e54f36@postgresql.org
Backpatch-through: 19

Branch
------
REL_19_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/747dbf30d14804bac400729a670ba3e3e88c7879

Modified Files
--------------
src/backend/executor/execGrouping.c | 35 +++++++++++++++++++++++++++--------
1 file changed, 27 insertions(+), 8 deletions(-)