Improper usage of MemoryContext in nodeSubplan.c for buildSubPlanHash() function. This maybe causes allocate memory failed.

Hi all,

This is a potential memory error in nodeSubplan.c or execGrouping.c
Using select '1'::TEXT IN ((SELECT '1'::NAME) UNION ALL SELECT '1'::NAME);
to reproduce this bug.

You may see the memory content that slot1's tts_values[0] point to before
and after the statement : MemoryContextReset(evalContext) of
execTuplesMatch() in PG version with --enable-cassert switch on. You will
find that the memory is set to 0x7f7f7f7f.

Here are my code analysis:
For the executor node SubPlanState, The node has a ExprContext named
'innerecontext' that is created by itself in function ExecInitSubPlan().
Also, the 'innerecontext' is used to build a project info, 'projRight', of
SubPlanState node.

When the SubPlanState node is executed, a hash table of subplan will be
built, so buildSubPlanHash() will be called. In buildSubPlanHash function,
'hashtable' of SubPlanState will be initialized by calling
BuildTupleHashTable(), and the code passes the
'ecxt_per_tuple_memory' of 'innerecontext' to BuildTupleHashTable() as the
'hashtable''s tempcxt.

At this point, we can conclude that the 'projRight' and 'hashtable''s
'tempcxt' of SubPlanState node are all using the same memory context, that
is 'innerecontext' in SubPlanState node. So:
1) The memory of all the fetched tuples from 'projRight' will be located in
'innerecontext' of SubPlanState node.
2) All other intermediate result that is located in 'hashtable''s tempcxt,
also in fact, will reside in the 'innerecontext' of SubPlanState node.

Now next:
In buildSubPlanHash(), we will fetch tuple from 'projRight' to fill the
hash table of SubPlanState node. As we known, all the fetched tuples are
located in the 'innerecontext'. When filling the tuple hash table, if the
tuple already exists in the hash table of SubPlanState node, the match
function execTuplesMatch() will be called by tuples matching, but in this
function, the statement:
MemoryContextReset(evalContext);
will be reset 'evalContext', to free some memory usage. In fact, the
'evalContext' is equal to 'innerecontext' that the fetched tuples are
located in, and actually this statement will reset the 'innerecontext'. So
the fetched tuple's memory are all lost. That's the problem.

In fact, this error only in debug version, but not in release version. In
debug using --enable-cassert to configure, the memory will be set to
0x7f7f7f7f, but not for release. For this error memory usage, the pg does
not always report error because the 0x7f7f7f in testing Macro
VARATT_IS_COMPRESSED() and VARATT_IS_EXTERNAL() are always false in
!WORDS_BIGENDIAN platform such as i386 GNU Linux and in WORDS_BIGENDIAN, the
testing Macro VARATT_IS_COMPRESSED() will be true and error may be reported
such as AIX(Power 6 AIX V6.1).

To fix this problem, we can use another memory context to passin
BuildTupleHashTable() as the hashtable's tempcxt, or use other memory
context as hash table's tempcxt or other ways.

Any questions, please contact me.
Regards