Re: How to estimate the shared memory size required for parallel scan?

On Thu, Aug 16, 2018 at 11:36 PM, Masayuki Takahashi
<masayuki038(at)gmail(dot)com> wrote:
> I am trying to change cstore_fdw to scan in parallel. FDW interface provide
> 'EstimateDSMForeignScan' for required shared memory size to scan in parallel.
>
> How to estimate the shared memory size required for parallel scan?

It's a slightly strange use of the word "estimate". It means "tell me
how much shared memory you need". Later, your
InitializeDSMForeignScan() callback will receive a pointer to exactly
that much shared memory to initialise. Then
InitializeWorkerForeignScan will also receive a pointer to that
memory, inside every worker process. Note that it may be mapped at a
different address in each process, so be careful not to use raw
pointers. It's up to you to design a struct to hold whatever data,
spinlocks, LWLocks, atomics etc you might need to orchestrate your
parallel scan.

It works much the same way for built-in executor nodes that are
parallel-aware by the way. For example, ExecHashJoinEstimate()
reserves sizeof(ParallelHashJoinState), and then in
ExecHashJoinInitializeDSM() it allocates and initialises it, and
ExecHashJoinIntializeWorker() tells the workers about it. The
built-in executor nodes have to do a little bit more work than FDWs,
using the plan node ID to allocate and look things up in a "TOC"
(table of contents), but nodeForeignScan.c does that work for you in
your case: it just asks you how much you want, and then gives it to
you.

--
Thomas Munro
http://www.enterprisedb.com