Re: Reducing relcache memory usage: deduping index shapes

Hi Matthias,

> This is a part of the work I discussed in my PGConf.DEV talk of this year [0].
>
> Many of Index's allocated fields in RelationData have the same (or
> practically the same) contents for many indexes. E.g. a btree index on
> a bigint column will always have the same contents in rd_opfamily,
> rd_opcintype, and rd_support, and (given the same opclass options in
> each column) will have equivalent rd_supportinfo.

I'm glad to see work on reducing relcache's memory footprint. I've seen
production workloads with 10-100k tables and many hundred thousand indexes.

> The attached patchset adds a deduplication layer into the relcache,
> which makes sure we only allocate one set of (rd_opfamily,
> rd_opcintype, rd_support, rd_supportinfo) for indexes with equivalent
> key definitions (so, a matching number of key attributes, opclasses,
> and AM).

Why did you specifically worked on deduplicating the index fields in
RelationData? Is that consuming most out of all of RelationData?

> Additionally, it includes a patch by Andres (polished by me) that adds
> a proxy context, which reduces the the overhead of small and
> long-lived allocations in (what we expect to be) small memory contexts
> by forwarding the allocations to malloc (after wrapping the struct).
>
> Earlier versions of the patch adjusted aset.c to accept smaller memory
> context sizes, but I abandoned that approach in favour of Andres'
> ProxyContext -- it can outsource most the complexities of memory
> management to the system allocator.
>
> Patches in this patchset:
> 0001/0002: prepare relcache for deduplication.
> 0003: implements the deduplication
> 0004: Andres' ProxyContext patch
> 0005: Use proxy context in relcache for 'index data'

On the testing end: if this effort makes more progress, I could run this
against the aforementioned database to get some real-world numbers.

One more micro optimization that we could do is better packing
RelationData. On my AMD64 system sizeof(RelationData) == 488 bytes. By
reordering the members we could get it down to 440 bytes which is about
10% savings.

--
David Geier