Re: compiler attribute alloc_size

From: "Tristan Partin" <tristan(at)partin(dot)io>
To: "Peter Eisentraut" <peter(at)eisentraut(dot)org>
Cc: "Andres Freund" <andres(at)anarazel(dot)de>, "pgsql-hackers" <pgsql-hackers(at)postgresql(dot)org>, "Zsolt Parragi" <zsolt(dot)parragi(at)percona(dot)com>
Subject: Re: compiler attribute alloc_size
Date: 2026-11-20 15:59:57
Message-ID: DN05VVAWZ6VA.1VWZIVSBPWUCC@partin.io
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Fri Nov 20, 2026 at 3:34 PM UTC, Peter Eisentraut wrote:
> On 16.11.26 18:49, Tristan Partin wrote:
>>> - TRGM in contrib/pg_trgm/ is an instance of what [0] talks about: It
>>> has a trailing flexible array member, but it also has trailing padding,
>>> so allocating palloc(TRGMHDRSIZE) causes a warning, because it's less
>>> than sizeof(TRGM). In my patch I work around this by allocating like
>>> palloc(Max(TRGMHDRSIZE, sizeof(TRGM))).
>>
>> Maybe we should add an allocator function or macro for this? In the
>> counted_by discussion, I talked a bit about how core does a pretty good
>> job of providing allocator helpers for structs with flexible array
>> members. I realize this is contrib/, but it might still be valuable.
>>
>>> - There is a bit of a maze of different structs and unions that underlie
>>> the numeric type. The allocation (Numeric) palloc(NUMERIC_HDRSZ_SHORT)
>>> gets flagged by the compiler. In my patch I change this to (Numeric)
>>> palloc(sizeof(struct NumericData)), which allocates a bit more but makes
>>> the sizes and types match. Maybe some more judicious reshuffling of the
>>> underlying data structures could avoid this, or maybe it's okay like this.
>>
>> Like the previous comment, I wonder if some helper allocator
>> functions/macros could be useful. But as you said, maybe some
>> reshuffling could avoid that all together, not sure.
>
> In the attached patch series, I have an idea for a more generalized
> approach: A macro that computes the required size and then a wrapper
> macro around it for allocating. This then can be applied to any
> structure with a flexible array member, and it eliminates some
> duplication with the offsetof calls and gets you the kind of type safety
> that palloc_object and palloc_array provide.
>
> I have also added a patch that makes some more uses of this macro.
> These are the same kinds of problems as the one I showed for pg_trgm and
> numeric, except that the size isn't constant and so the compiler won't
> flag it.

Overall, I think the concept looks good! Maybe adding
a palloc_empty_flex_array() macro could be useful to make the
0 allocations a little bit nicer to read.

> @@ -1226,6 +1226,24 @@ pg_noreturn extern void ExceptionalCondition(const char *conditionName,
> */
> #define Min(x, y) ((x) < (y) ? (x) : (y))
>
> +/*
> + * flex_size
> + * Return the size of a struct "type" whose flexible array member
> + * "member" holds "count" elements.
> + *
> + * The result is never less than sizeof(type), so that any trailing padding
> + * of the struct is covered. "count" is evaluated only once.
> + */
> +#define flex_size(type, member, count) \
> + flex_size_impl(sizeof(type), offsetof(type, member), \
> + sizeof(((type *) 0)->member[0]), (count))
> +
> +static inline size_t
> +flex_size_impl(size_t structsz, size_t offset, size_t elemsz, size_t count)
> +{
> + return Max(structsz, offset + count * elemsz);
> +}
> +
>
> /* Get a bit mask of the bits set in non-size_t aligned addresses */
> #define SIZE_T_ALIGN_MASK (sizeof(size_t) - 1)

It might behoove us to use the same safe math techniques that we use in
palloc_mul() here. Maybe I am being too cautious. It is not clear to me
if any potential "count" values would ever be derived from user input.
It would protect from us _succeeding_ to allocate a potentially smaller
amount of memory, and then writing or reading past the end.

>>> - As of commit 6a3da13d376 from this week, FormData_pg_attribute has 4
>>> bytes of trailing padding, and so sizeof(FormData_pg_attribute) is now
>>> different from ATTRIBUTE_FIXED_PART_SIZE. This causes a warning in one
>>> place. We had a discussion about this some time ago [1], where the
>>> opinions diverged between like, it's important to keep this distinction
>>> in case we ever have padding bytes, and, it's already incorrect and
>>> would be even more incorrect if we had padding bytes. ;-) Well, we're
>>> here now, so perhaps this should be revisited.
>
> I have adjusted the description of this patch a bit. There are a lot of
> details to be figured out in this area, but at least we can fix this
> particular issue easily and in a consistent way. tupdesc.c already uses
> sizeof(FormData_pg_attribute) consistently for allocating pg_attribute
> structures. It only uses ATTRIBUTE_FIXED_PART_SIZE for things like
> memcpy(). With the proposed patch, we make bootstrap.c consistent in
> this regard, and we make the warning go away. The deeper issues can
> then be researched independent of this.

I agree that this problem is likely orthogonal, and we can move it to
a separate discussion.

>> In [1], Tom said:
>>
>>> What I'm trying to figure out is if we need to put in place some
>>> provisions to prevent there from being pad bytes at the end of any
>>> catalog struct.
>>
>> I figured I could propose at least one way to flag padding bytes in the
>> attached patch. I am currently not proposing a solution.
>>
>> Two options proposed in the thread[1]:
>>
>> - Placeholder fields exactly matching the size of the padding bytes.
>>
>> #ifdef CATALOG_VARLEN /* variable-length fields start here */
>> pg_node_tree ev_qual;
>> pg_node_tree ev_action;
>> #else
>> uint8 dummy;
>> #endif
>>
>> - Use different structs for on-media and in-memory representations.
>>
>> To not rock the boat too much, maybe I would throw my hat in the ring
>> for option 1? It seems like the simplest and most immediate solution.
>
> I think we could get the same check simpler and more portably by having
> genbki.pl generate some static assertions that check like
>
> static_assert(sizeof(FormData_pg_attribute) ==
> offsetof(FormData_pg_attribute, attcollation) +
> sizeof(FormData_pg_attribute.attcollation));
>
> But this flags a bunch of issues. Sorting that out might be a big project.

I had a similar idea as well.

Overall, I think that the non-WIP patches look good, but it would
probably be useful to get another committer's thoughts before moving
forward.

--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Robert Haas 2026-11-20 16:20:27 Re: log_postmaster_stats
Previous Message Rui Zhao 2026-11-20 15:47:45 Re: [PATCH v2] Fix SP-GiST index-only scans that return only INCLUDE columns