GiST API Adancement

Hi hackers,

I want to hack a little. I applied for GSoC project (https://wiki.postgresql.org/wiki/GSoC_2017#GiST_API_advancement), but now I'm going to hack on my own. With the help of Andrew Borodin, I want to start the project with adding a third state to collision check. The third state is that:
subtree is totally within the query. In this case, GiST scan can scan down subtree without key checks.

After reading some code, I get this plan to modify the code:

1. Modify the consistent function of datatypes supported by GiST.

1.1 Start with cube, g_cube_consistent should return a third state when calling g_cube_internal_consistent. Inspired by Andrew Borodin, I have two solutions, 1)modify return value, 2) modify a reference type parameter.

2. Modify the gistindex_keytest in gistget.c to return multiple states

2.1 Need declare a new enum type(or define values) in gist_private.h or somewhere

3. Modify the gitsScanPage in gistget.c to deal with the extra state

4. Add a state to mark the nodes under this GISTSearchItem are all with in the query

4.1 Two ways to do this: 1) add a state to GISTSearchItem (prefered) 2) Somewhere else to record all this kind of items

4.2 We may use the block number as the key

4.3 Next time when the gistScanPage met this item, just find all the leaves directly(need a new function)

After this, I shall start to benchmark the improvement and edit the code of other datatypes.

Hope you hackers can give me some suggestions~

--
Dong