Re: improving speed of make check-world

From: Peter Eisentraut <peter_e(at)gmx(dot)net>
To: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Cc: Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr>
Subject: Re: improving speed of make check-world
Date: 2015-02-15 02:01:48
Message-ID: 54DFFE0C.4060100@gmx.net
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

On 8/31/14 5:36 AM, Fabien COELHO wrote:
> Running "make -j2 check-world" does not work because "initdb" is not
> found by "pg_regress". but "make -j1 check-world" does work fine. It
> seems that some dependencies might be missing and there is a race
> condition between temporary install and running some checks?? Maybe it
> is not expected to work anyway? See below suggestions to make it work.

Here is an updated patch that fixes this problem.

The previous problem was simply a case were the make rules were not
parallel-safe.

For recursive make, we (through magic) set up targets like this:

check: check-subdir1-check check-subdir2-check

And with my old patch we added

check: temp-install

So the aggregate prerequisites were in effect something like

check: check-subdir1-check check-subdir2-check temp-install

And so there was nothing stopping a parallel make to descend into the
subdirectories before the temp install was set up.

What we need is additional prerequisites like

check-subdir1-check check-subdir2-check: temp-install

I have hacked this directly into the $(recurse) function, which is ugly.
This could possibly be improved somehow, but the effect would be the
same in any case.

With this, I can now run things like

make -C src/pl check -j3
make -C src/bin check -j8

A full make check-world -j$X is still prone to fail because some test
suites can't run in parallel with others, but that's a separate problem
to fix.

Note: We have in the meantime added logic to pg_regress to clean up the
temporary installation after the run. This changes that because
pg_regress is no longer responsible for the temporary installation.
pg_regress still cleans up the temporary data directory, so you still
get quite a bit of space savings. But the temporary installation is not
cleaned up. But since we would now only use a single temporary
installation, the disk space usage still stays in the same order of
magnitude.

Attachment Content-Type Size
regress-temp-instance-v3.patch text/x-patch 31.0 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tom Lane 2015-02-15 02:09:39 Re: improving speed of make check-world
Previous Message Tom Lane 2015-02-15 01:55:39 Re: Allow "snapshot too old" error, to prevent bloat