Re: How to estimate the shared memory size required for parallel scan?

On Mon, Aug 20, 2018 at 11:09 AM Thomas Munro
<thomas(dot)munro(at)enterprisedb(dot)com> wrote:
> 2. Teach your GetForeignPath function to do something like this:
> [blah blah blah]

I was pinged off-list by someone who is working on a parallel-aware
FDW, who asked if I still had the test code I mentioned up-thread.
While digging that out, I couldn't resist hacking it a bit more until
it gave the right answers, only sooner:

$ seq -f '%20.0f' 1 10000000 > numbers.csv

create extension file_fdw;
create server files foreign data wrapper file_fdw;
create foreign table numbers (n int) server files
options (filename '/path/to/numbers.csv', format 'csv');
explain select count(*) from numbers;

select count(*) from numbers;

Non-parallel: 2.6s
1 worker: 1.4s
2 workers: 0.9s
3 workers: 0.7s

Finally, I can do parallel hash joins between CSV files!

select count(*) from numbers n1 join numbers n2 using (n);

Non-parallel: 11.4s
1 worker: 6.6s
2 workers: 4.8s
3 workers: 4.1s

There are probably some weird fence-post or accounting bugs hiding in
this patch -- it has to count bytes carefully, and deal with some edge
cases around lines that span chunks. It's only a rough draft, but
might eventually serve as a useful example of a parallel-aware FDW.

--
Thomas Munro
https://enterprisedb.com