Re: Batching in executor

Hi Mats,

Thanks for the feedback.

First, I would like to clarify the main point. Do you mean that
filters and aggregates would run inside the scan TTS? Or would
separate plan nodes pass native batches between them?

> … requiring a conversion to PostgreSQL-native columnar format
> would cost CPU and potentially memory.

I do not propose converting a native batch after the scan. A batch
can pass between nodes that understand its batch->ops. If a parent
does not understand the format, it can materialize only the selected
rows and needed columns as Datums.

For example, ColumnarScan and a custom batch-aware Agg node could
exchange TimescaleDB batches without conversion. Both nodes would
remain visible in the plan.

> I think it would be more generic to "push down" the operations (for example,
> the filters and aggregates) into the TTS and let the TTS implementor (under
> control by the TAM implemetor) apply the operator to the batch.

I am worried that this could turn TTS into a hidden executor. If TTS
executes aggregates, it may later need to execute sorts and joins too.

I think TTS should manage the batch format, lifetime, selection,
column access, and Datum materialization. Filters, aggregates, sorts,
and joins should remain plan nodes. They may be CustomScan nodes that
understand a specific native format.

> It would be easier and less intrusive to just modify the "getsomeattr" to
> accept a Bitmapset instead of an integer.

An optional getcolumnattr looks useful. If it fills non-adjacent
columns, we need to record which columns are ready, because tts_nvalid
can only describe the first N columns.

I still think filter columns and output columns should be listed
separately. This lets the scan read output columns only for rows that
pass the filters.

> * Add a function to advance to the next row, which would then do filtering
> and aggregation internally and return the resulting "next" row. If the advance
> function returns "false", it calls ExecProcNode to get a new batch.

An advance callback also looks useful. It should only move within the
current batch. The plan node should fetch the next batch and update
executor statistics.

Best regards,
Denis Smirnov