Re: Parallel make problem with git master

Excerpts from Robert Haas's message of mar mar 08 10:38:29 -0300 2011:
> On Mon, Mar 7, 2011 at 10:28 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > I wrote:
> >> I think what is happening here is that make launches concurrent sub-jobs
> >> to do "make install" in each of interfaces/libpq and interfaces/ecpg,
> >> and the latter launches a sub-sub-job to do "make all" in
> >> interfaces/libpq, and make has no idea that these are duplicate sub-jobs
> >> so it actually tries to run both concurrently.  Whereupon you get all
> >> sorts of fun failures.  I'm not sure if there is any cure that's not
> >> worse than the disease.
> >
> > BTW, how many people here have read "Recursive Make Considered Harmful"?
> >
> > http://aegis.sourceforge.net/auug97.pdf
> >
> > Because what we're presently doing looks mighty similar to what he's
> > saying doesn't work and can't be made to work.

Yeah, I read it some years ago and considered it, but it was too
disruptive or I was too new here, maybe both :-)

The bit I looked at, at the time, was src/backend/mb/conversion_procs,
because that was where the biggest hit on parallelization was taken (a
single lib at a time -- the real time CPU usage chart clearly showed the
problem. Not sure if that's still a problem).

> I'm not sure whether it makes sense to go that far or not. But I
> think it'd make sense to at least try this for the backend. It does
> seem pretty silly to have a Makefile in every single directory.

We already do that for the backend. Not exactly a single Makefile, but
the dependencies are all declared in indirectly in src/backend/Makefile
with the common.mk tricks.

Where it doesn't work is in the other subdirs, c.f. the current problem
with interfaces/libpq and interfaces/ecpg. It would be a lot more
difficult to fix there, I think, but maybe I'm wrong.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support