Re: Extracting only the columns needed for a query

On Fri, Jan 31, 2020 at 9:52 AM Melanie Plageman <melanieplageman(at)gmail(dot)com>
wrote:

> I'm bumping this to the next commitfest because I haven't had a chance
> to address the questions posed by Dmitry. I'm sure I'll get to it in
> the next few weeks.
>
> > I believe it would be beneficial to add this potential API extension
> patch into
> > the thread (as an example of an interface defining how scanCols could be
> used)
> > and review them together.
>
> As for including some code that uses the scanCols, after discussing
> off-list with a few folks, there are three options I would like to
> pursue for doing this.
>
> One option I will pursue is using the scanCols to inform the columns
> needed to be spilled for memory-bounded hashagg (mentioned by Jeff
> here [1]).
>
>
> The third is exercising it with a test only but providing an example
> of how a table AM API user like Zedstore uses the columns during
> planning.
>

Outside of the use case that Pengzhou has provided in [1], we started
looking into using scanCols for extracting the subset of columns
needed in two cases:

1) columns required to be spilled for memory-bounded hashagg
2) referenced CTE columns which must be materialized into tuplestore

However, implementing these optimization with the scanCols patch
wouldn't work with its current implementation.

The scanCols are extracted from PlannerInfo->simple_rel_array and
PlannerInfo->simple_rte_array, at which point, we have no way of
knowing if the column was aggregated or if it was part of a CTE or
anything else about how it is used in the query.

We could solve this by creating multiple bitmaps at the time that we
create the scanCols field -- one for aggregated columns, one for
unaggregated columns, one for CTEs. However, that seems like it would
add a lot of extra complexity to the common code path during planning.

Basically, scanCols are simply columns that need to be scanned. It is
probably okay if it is only used by table access method API users, as
Pengzhou's patch illustrates.

Given that we have addressed the feedback about showing a use case,
this patch is probably ready for a once over from Dmitry again. (It is
registered for the March fest).

[1]
https://www.postgresql.org/message-id/CAG4reAQc9vYdmQXh%3D1D789x8XJ%3DgEkV%2BE%2BfT9%2Bs9tOWDXX3L9Q%40mail.gmail.com

--
Melanie Plageman