Fix "could not find memoization table entry"

Hi,

Recently, I encountered an error: could not find memoization table
entry on TPCH(S=1) test.
The query was as follows:
psql (19devel)
Type "help" for help.

postgres=# SELECT
FROM partsupp AS ref_0,
LATERAL (SELECT ref_0.ps_suppkey AS c3,
(SELECT n_regionkey
FROM nation
LIMIT 1) AS c4
LIMIT ALL) AS subq_0
WHERE hash_numeric(ref_0.ps_supplycost) = subq_0.c4;
ERROR: could not find memoization table entry
postgres=# explain SELECT
FROM partsupp AS ref_0,
LATERAL (SELECT ref_0.ps_suppkey AS c3,
(SELECT n_regionkey
FROM nation
LIMIT 1) AS c4
LIMIT ALL) AS subq_0
WHERE hash_numeric(ref_0.ps_supplycost) = subq_0.c4;
QUERY PLAN
-----------------------------------------------------------------------------------------
Nested Loop (cost=0.06..57942.04 rows=4000 width=0)
-> Seq Scan on partsupp ref_0 (cost=0.00..25560.00 rows=800000 width=10)
-> Memoize (cost=0.06..0.09 rows=1 width=4)
Cache Key: hash_numeric(ref_0.ps_supplycost), ref_0.ps_suppkey
Cache Mode: binary
Estimates: capacity=80659 distinct keys=88915 lookups=800000
hit percent=80.63%
-> Subquery Scan on subq_0 (cost=0.05..0.08 rows=1 width=4)
Filter: (hash_numeric(ref_0.ps_supplycost) = subq_0.c4)
-> Result (cost=0.05..0.06 rows=1 width=8)
InitPlan expr_1
-> Limit (cost=0.00..0.05 rows=1 width=4)
-> Seq Scan on nation (cost=0.00..1.25
rows=25 width=4)
(12 rows)

The hash_numeric result type is int. If I forced binary_mode to
logical, there was no error anymore.
So I think this may be a bug.

How to easily reproduce:
1. prepare tpch(s=1) data
2. Using gdb to set the mstate->mem_limit to 170 after the first tuple
was put into the cache in func cache_lookup()
3. When putting the second tuple into the cache, the mem_used will
exceed the mem_limit, so
calling the cache_reduce_memory() to remove the first tuple. But it
cannot find the first tuple in the hash table.

I did some research about this issue. When we insert the first tuple
into the cache, the first column(hash_numeric(ref_0.ps_supplycost))
value is:
(gdb) p /x pslot->tts_values[i]
$2 = 0xaf27c7c7

(gdb) p hkey
$2 = 38469220

But in the cache_reduce_memory(), its value is like this:
(gdb) p /x pslot->tts_values[i]
$5 = 0xffffffffaf27c7c7

(gdb) p hkey
$7 = 288723292

The hkeys returned by datum_image_hash() are different, so we can't
find the entry in the hash table.

In the datum_image_hash(), if typByVal is true, calling
result = hash_bytes((unsigned char *) &value, sizeof(Datum));

I think we should use typLen here, not sizeof(Datum).
I tried this way and didn't encounter any errors again.

I added David to the cc list. He may know more about this module.

--
Thanks,
Tender Wang