Re: Support Parallel Query Execution in Executor

On 4/9/06, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I was considering a variant idea in the shower this morning:suppose
> that we invent one or more "background reader" processes that have
> basically the same infrastructure as the background writer, but have
> the responsibility of causing buffer reads to happen at useful times
> (whereas the writer causes writes to happen at useful times). The
> idea would be for backends to signal the readers when they know they
> will need a given block soon, and then hopefully when they need it
> it'll already be in shared buffers.

This is sort of what I'm playing with. There are N-number of backends
which are configured at startup and are available solely for parallel
processing. If you have parallelism enabled and you have a query
which the planner says can take advantage of it, your backend becomes
a sort of "coordinator" and signals the number of backends defined by
(right now) a global parallelism degree parameter for participation in
query execution.

> The major issues I can see are:
>
> 1. We'd need a shared-memory queue of read requests, probably much like
> the queue of fsync requests. We've already seen problems with
> contention for the fsync queue, IIRC, and that's used much less heavily
> than the read request queue would be. So there might be some
> performance issues with getting the block requests sent over to the
> readers.

Yes, I was looking at this contention as well. However, I haven't
looked at the fsync request queue in detail, but I'm going to play
around with a simple initial semaphore test such that if the queue is
full, the query will just be processed serially. I'll also take a
look at the fsync queue and see how it's working.

While it's down the road a ways, I'm really looking forward to seeing
parallel index builds, sorts, and joins. In my opinion, all of our
combined ideas and system knowledge is really going to pay off big
time on this one.

--
Jonah H. Harris, Database Internals Architect
EnterpriseDB Corporation
732.331.1324