Grouping Sets: Fix unrecognized node type bug

Hi,

It looks like we do support nested GROUPING SETS, I mean Sets withing
Sets, not other types. However this nesting is broken.

Here is the simple example where I would expect three rows in the
result. But unfortunately it is giving "unrecognized node type"
error. Which is something weird and need a fix.

postgres=# create table gstest2 (a integer, b integer, c integer);
postgres=# insert into gstest2 values (1,1,1), (1,1,1), (1,1,1),
(1,1,1), (1,1,1), (1,1,1), (1,1,2), (1,2,2), (2,2,2);
postgres=# select sum(c) from gstest2
group by grouping sets((), grouping sets((), grouping sets(())))
order by 1 desc;
ERROR: unrecognized node type: 926

I spend much time to understand the cause and was looking into
transformGroupingSet() and transformGroupClauseList() function.
I have tried fixing "unrecognized node type: 926" error there,
but later it is failing with "unrecognized node type: 656".

Later I have realized that we have actually have an issue while
flattening grouping sets. If we have nested grouping sets like
above, then we are getting GroupingSet node inside the list and
transformGroupClauseList() does not expect that and end up with
this error.

I have tried fixing this issue in flatten_grouping_sets(), after
flattening grouping sets node, we need to concat the result with
the existing list and should not append. This alone does not
solve the issue as we need a list when we have ROW expression.
Thus there, if not top level, I am creating a list now.

Attached patch with few testcases too.

Please have a look.

Thanks

--
Jeevan B Chalke
Principal Software Engineer, Product Development
EnterpriseDB Corporation
The Enterprise PostgreSQL Company