Skip site navigation (1) Skip section navigation (2)

Re: Multithread Query Planner

From: "Pierre C" <lists(at)peufeu(dot)com>
To: "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>, "Robert Haas" <robertmhaas(at)gmail(dot)com>
Cc: "Merlin Moncure" <mmoncure(at)gmail(dot)com>, "Christopher Browne" <cbbrowne(at)gmail(dot)com>, Frederico <zepfred(at)gmail(dot)com>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Multithread Query Planner
Date: 2012-01-27 19:56:31
Message-ID: op.v8rdoheaeorkce@apollo13 (view raw or flat)
Thread:
Lists: pgsql-hackers
>> Not to mention palloc, another extremely fundamental and non-reentrant
>> subsystem.
>>
>> Possibly we could work on making all that stuff re-entrant, but it would
>> be a huge amount of work for a distant and uncertain payoff.

> Right.  I think it makes more sense to try to get parallelism working
> first with the infrastructure we have.  Converting to use threading,
> if we ever do it at all, should be something we view as a later
> performance optimization.  But I suspect we won't want to do it
> anyway; I think there will be easier ways to get where we want to be.

Multithreading got fashionable with the arrival of the Dual-core CPU a few  
years ago. However, multithreading as it is used currently has a huge  
problem : usually, threads share all of their memory. This opens the door  
to an infinite number of hard to find bugs, and more importantly, defeats  
the purpose.

"Re-entrant palloc()" is nonsense. Suppose you can make a reentrant  
palloc() which scales OK at 2 threads thanks to a cleverly placed atomic  
instruction. How is it going to scale on 64 cores ? On HP's new 1000-core  
ARM server with non-uniform memory access ? Probably it would suck very  
very badly... not to mention the horror of multithreaded exception-safe  
deallocation when 1 thread among many blows up on an error...

For the ultimate in parallelism, ask a FPGA guy. Is he using shared memory  
to wire together his 12000 DSP blocks ? Nope, he's using isolated  
Processes which share nothing and communicate through FIFOs and hardware  
message passing. Like shell pipes, basically. Or Erlang.

Good parallelism = reduce shared state and communicate through  
data/message channels.

Shared-everything multithreading is going to be in a lot of trouble on  
future many-core machines. Incidentally, Postgres, with its Processes,  
sharing only what is needed, has a good head start...

With more and more cores coming, you guys are going to have to fight to  
reduce the quantity of shared state between processes, not augment it by  
using shared memory threads !...

Say you want to parallelize sorting.
Sorting is a black-box with one input data pipe and one output data pipe.
Data pipes are good for parallelism, just like FIFOs. FPGA guys love black  
boxes with FIFOs between them.

Say you manage to send tuples through a FIFO like zeromq. Now you can even  
run the sort on another machine and allow it to use all the RAM if you  
like. Now split the black box in two black boxes (qsort and merge),  
instanciate as many qsort boxes as necessary, and connect that together  
with pipes. Run some boxes on some of this machine's cores, some other  
boxes on another machine, etc. That would be very flexible (and scalable).

Of course the black box has a small backdoor : some comparison functions  
can access shared state, which is basically *the* issue (not reentrant  
stuff, which you do not need).

In response to

Responses

pgsql-hackers by date

Next:From: Peter GeogheganDate: 2012-01-27 20:33:56
Subject: Re: Progress on fast path sorting, btree index creation time
Previous:From: Sergey KonoplevDate: 2012-01-27 19:42:46
Subject: Re: pg_statistic, lack of documentation

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group