Re: How to estimate the shared memory size required for parallel scan?

(Sorry, once I sent to Thomas only. This is re-post.)

Hi Thomas,

Thanks you for excellent explaining about shared memory in parallel
scan and 'foreign path'.
Those are points that I want to know. thanks.

> If you just supply an IsForeignScanParallelSafe function that returns
> true, that would allow your FDW to be used inside parallel workers and
> wouldn't need any extra shared memory, but it wouldn't be a "parallel
> scan". It would just be "parallel safe". Each process that does a
> scan of your FDW would expect a full normal scan (presumably returning
> the same tuples in each process).

I think that parallel scan mechanism uses this each worker's full
normal scan to partitioned records, right?
For example, I turned IsForeignScanParallelSafe to true in cstore_fdw
and compared partitioned/non-partitioned scan.

https://gist.github.com/masayuki038/daa63a21f8c16ffa8138b50db9129ced

This shows that counted by each partition and 'Gather Merge' merge results.
As a result, parallel scan and aggregation shows the correct count.

Then, in the case of cstore_fdw, it may not be necessary to reserve
the shared memory in EstimateDSMForeignScan.

> So I guess this hasn't been done before and would require some more
> research.

I agree. I will try some query patterns.
thanks.
2018年8月18日(土) 23:08 Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com>:
>
> On Sun, Aug 19, 2018 at 1:40 AM, Thomas Munro
> <thomas(dot)munro(at)enterprisedb(dot)com> wrote:
> > A true parallel scan of an FDW would be one where each process emits
> > an arbitrary fraction of the tuples, but together they emit all of the
> > tuples. You'd almost certainly need to use some shared memory to
> > coordinate that. To say that you support that, I think your
> > GetForeignPaths() function would need to call add_partial_path(). And
> > unless I'm mistaken, whether or not InitializeDSMForeignScan etc are
> > called might be the only indication you get of whether you need to run
> > in parallel-aware mode. I haven't personally heard of any FDWs that
> > can do this yet, but I just tried hacking file_fdw to register a
> > partial path and it seems to work (though of course the results are
> > duplicated because the emitted tuples are not actually partial).
>
> ... though I just noticed that my quick test used "Single Copy" mode.
> I think I see why: it looks like core's create_foreignscan_path()
> function might need to take num_workers and set parallel_aware if > 0.
> So I guess this hasn't been done before and would require some more
> research.
>
> --
> Thomas Munro
> http://www.enterprisedb.com

--
高橋 真之