Re: Improve hash aggregate spilling by writing only the needed columns

Hi Mario,

Thanks for working on this, it is a welcome improvement for analytics
workloads.

On Fri, Nov 6, 2026 at 2:41 PM Mario Karuza <mkaruza(dot)pg(at)icloud(dot)com> wrote:
> Attached is v2, rebased on current master (4a934833fb0):

I tested v2 on top of 4a934833fb0 (Linux x86_64, gcc 14, LLVM 19).

> 1. Fixed the JIT crash reported by Xiaoxuan.

The crash reproduces with v1 and is fixed in v2. The regression tests pass, also
with JIT forced for every query, and the cases Ethan listed (NULLs, by-ref
types, recursive respill, rescans, grouping sets) return the same results as a
reference run, with and without JIT.

> 2. Used Ethan lazy initialization.

I see no measurable overhead when nothing spills, neither on an empty table nor
with a work_mem large enough to avoid spilling.

I reproduced your TPC-H Q18 and edge-case numbers, then ran ClickBench [1]
(official 105-column hits table, first 5M rows, the 43 official queries) with
the default configuration (work_mem = 4MB, no parallel workers), 6 runs per
query, median:

master v2
q13 GROUP BY SearchPhrase 1964 ms 1318 ms -32.9%
q16 GROUP BY UserID 4712 ms 1716 ms -63.6%
q17 GROUP BY UserID, SearchPhrase 5551 ms 2317 ms -58.3%
q31 GROUP BY SearchEngineID, ClientIP 1838 ms 1296 ms -29.5%
q33 GROUP BY WatchID, ClientIP 6228 ms 3651 ms -41.4%
q34 GROUP BY URL 8764 ms 3104 ms -64.6%

Geometric mean: -38% on the 14 queries whose HashAggregate spills, -1.7% (within
noise) on the 29 others, -15.6% overall.

The 4 spilling queries that group on expressions (q19, q29, q36, q40) do not
change: the scan already projects, so there is nothing to remove. That might be
worth a sentence in the commit message.

[1] https://github.com/ClickHouse/ClickBench

Regards,
Alain