Reducing System Allocator Thrashing of ExecutorState to Alleviate FDW-related Performance Degradations

Hi everyone,

I've been working on a federated database project that heavily relies on
foreign data wrappers. During benchmarking, we noticed high system CPU
usage in OLTP-related cases, which we traced back to multiple brk calls
resulting from block frees in AllocSetReset upon ExecutorEnd's
FreeExecutorState. This is because FDWs allocate their own derived
execution states and required data structures within this context,
exceeding the initial 8K allocation, that need to be cleaned-up.

Increasing the default query context allocation from ALLOCSET_DEFAULT_SIZES
to a larger initial "appropriate size" solved the issue and almost doubled
the throughput. However, the "appropriate size" is workload and
implementation dependent, so making it configurable may be better than
increasing the defaults, which would negatively impact users (memory-wise)
who aren't encountering this scenario.

I have a patch to make it configurable, but before submitting it, I wanted
to hear your thoughts and feedback on this and any other alternative ideas
you may have.

--
Jonah H. Harris