Re: About Custom Aggregates, C Extensions and Memory

Marthin Laubscher <postgres(at)lobeshare(dot)co(dot)za> writes:
> I was fixing on (ab)using user-defined aggregates for a concept that would, on second thought, be better described as a key-only table persisted and manipulated as a single opaque value that is only ever expanded to a table/set/collection in memory. Initial focus is on storing 8-byte integer IDs but bigger UUIDs might be needed down the line.

Hm. We do not have in-memory tables, although in some cases a
temporary table is close enough. But there is one other pre-existing
mechanism that might help you: "expanded objects". The idea there is
that you have some "flat" representation of your data type that can
go into a table at need, but you also have an in-memory representation
that is better suited to computation, and most of your complicated
operations prefer to work on the expanded representation. The name of
the game then becomes how to minimize the number of times a value gets
flattened and expanded as you push it around in a computation.
As of v18, we have a pretty decent story for that when it comes to
values you are manipulating within pl/pgsql functions, although less
so if you need to use other languages. (The expanded-object
functionality exists before v18, but it's hard for user-defined types
to avoid excess flattening in earlier versions.)

If that sounds promising, see

src/include/utils/expandeddatum.h
src/backend/utils/adt/expandeddatum.c

as well as the two in-core uses of the concept:

src/backend/utils/adt/array_expanded.c
src/backend/utils/adt/expandedrecord.c

This thread that led to the v18 improvements might be useful too:

https://www.postgresql.org/message-id/flat/CACxu%3DvJaKFNsYxooSnW1wEgsAO5u_v1XYBacfVJ14wgJV_PYeg%40mail.gmail.com

regards, tom lane