Re: BUG #19636: heap_fill_tuple misses short varlena conversion after EOH_flatten_into, causing 3-byte waste per row

пт, 21 авг. 2026 г. в 19:37, Andrey Rachitskiy <pl0h0yp1(at)gmail(dot)com>:

> History
> -------
> Short headers date from 3e23b68dac0 (2007, "Support varlena fields with
> single-byte headers and unaligned storage"). heap_form_tuple has packed
> inline 4-byte-header varlenas via VARATT_CAN_MAKE_SHORT ever since the
> modern heap_form_tuple path.
>
> Expanded objects arrived in 1dc5ebc9077 (2015, "Support expanded
> objects, particularly arrays, for better performance"). That commit
> added an EXTERNAL_EXPANDED branch in heap_compute_data_size / fill_val
> that flattens with EOH_flatten_into and writes the result as-is.
> Flatteners are required to produce an inline 4-byte-header varlena
> (see expandeddatum.h). The new branch never applied the existing
> short-header conversion afterward. So this looks like an omission from
> day one of expanded objects, not a later regression.
>
> Proposal Fix
> ---
> In those EXTERNAL_EXPANDED arms, if the attribute is packable and the
> flat size from EOH_get_flat_size() would fit a short header, flatten
> into a temporary palloc buffer and emit the short form into the tuple.
> Otherwise keep the previous path (align and flatten straight into the
> tuple).
>
> VARATT_CAN_MAKE_SHORT cannot be used on the expanded toast pointer
> itself. It requires a 4B_U varlena. Before flattening we only have
> the flat size, so the patch uses a size-only helper matching that
> macro's length rule.
>
> The temporary buffer is needed because EOH_flatten_into expects a
> maxaligned destination (same constraint as datumSerialize), while short
> packing writes at an unaligned data pointer. The temp is not pfree'd.
> fill_val can run with CurrentMemoryContext as a BumpContext. On
> master / REL_19, RecursiveUnion UNION DISTINCT stores hashed tuples in
> a BumpContext tuplescxt (c106ef08071, "Use BumpContext contexts in
> TupleHashTables"). LookupTupleHashEntry switches to that context
> before ExecCopySlotMinimalTupleExtra → heap_form_minimal_tuple. Bump
> does not support pfree. A variant that pfree'd the temp failed the
> "with" regress test on 19/master with:
>
> ERROR: pfree is not supported by the bump memory allocator
>
> REL_18 still used AllocSet for that context, so the same pfree passed
> there. The short-packable flat size is at most about 130 bytes. The
> chunk is reclaimed when the context is reset.
>
> Alternatives considered: Stack buffer was considered (fits the size
> bound), but palloc with EOH_flatten_into seemed better to me.
>
>
A follow-up on the repro and on the regress in v1.

The heaptuple omission is real, but the reporter's one-shot example often
does not reach it. With the default plan_cache_mode = auto, the first
executions of the PL/pgSQL INSERT use a custom plan. The bound array
parameter is substituted during planning. datumCopy flattens the
expanded object there, so fill_val later sees an ordinary 4-byte-header
varlena and packs it to short as usual. Both rows then show
pg_column_size 33 even without the fix.

I checked both cases with gdb on an unpatched build, breaking on
EOH_flatten_into.

1. Default plan_cache_mode (auto), one-shot PL/pgSQL INSERT — custom plan

EOH_flatten_into is called once, with allocated_size 36, from datumCopy
during planning (parameter substitution in eval_const_expressions):

#0 EOH_flatten_into (... allocated_size=36)
#1 datumCopy
#3 eval_const_expressions_mutator
...
# BuildCachedPlan / GetCachedPlan
# SPI_execute_plan_with_paramlist
# plpgsql exec_stmt_execsql

There is no fill_val frame on that hit. heap_form_tuple later sees an
ordinary flat 4-byte-header varlena and applies VARATT_CAN_MAKE_SHORT as
usual. pg_column_size is 33 for the plain INSERT and 33 for the
PL/pgSQL INSERT.

2. Same SQL with a generic plan

SET plan_cache_mode = force_generic_plan;

EOH_flatten_into is again called with allocated_size 36, but the caller
is the EXTERNAL_EXPANDED arm in fill_val:

#0 EOH_flatten_into (... allocated_size=36)
#1 fill_val (... heaptuple.c)
#2 heap_fill_tuple
#3 heap_form_tuple
...
# ExecModifyTable / SPI / plpgsql

That arm writes the flattener output as-is, so the PL/pgSQL row keeps
the 4-byte header. Without the fix pg_column_size is 33 (plain INSERT)
and 36 (PL/pgSQL). With the fix both are 33.

Under plan_cache_mode = auto the same fill_val path appears after the
plan cache switches from custom to generic (after several executions of
the INSERT). force_generic_plan just makes that path reliable on the
first call.

v2 of the patch is attached. The heaptuple change is the same as v1.
The plpgsql regress now sets force_generic_plan around the test so case
2 is what we cover. Case 1 would pass even without the fix.

--
Regards,
Rachitskiy Andrey