Re: Catcache rehash hits MaxAllocSize (1GB) and errors with "invalid memory alloc request size" on very large numbers of relations

Reshmithaa <reshmithaa(dot)b(at)zohocorp(dot)com> writes:
> While running ANALYZE across a database with 500,000+ tables (~350
> columns each) in a single client session, we hit with an error:
>   ERROR: invalid memory alloc request size 1073741824

Hmm...

> 1. Have RehashCatCache()/RehashCatCacheLists() allocate the bucket array
>    with MemoryContextAllocHuge() instead of MemoryContextAllocZero().
>    Is there a known reason this wasn't already done, or would a patch
>    along these lines be welcome?

That by itself wouldn't move the goalposts very far: you're within
a factor of 10 or so of overflowing the int32 fields like cc_ntups,
cc_nbuckets, etc. Widening those would take a lot of work, eg
it bleeds into hash calculations, not to mention that 32-bit
hash values for entries start to seem not wide enough when you
have that many entries. And I don't even want to think about
how much RAM a backend is going to be eating with 500K relcache
entries and corresponding catcache entries for a lot of them.

What would probably be a more productive answer is to look at limiting
the caches' size by discarding not-recently-used entries. We actually
had code to do that for catcaches, long ago, and dropped it because
keeping track of what was least recently used added large amounts of
overhead. But possibly a less-naive implementation could do better on
that score.

In the meantime, I think you've chosen an unworkable database setup
and you need to rethink how many tables you've got.

regards, tom lane