WIP patch for avoiding duplicate initdb runs during "make check"

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: pgsql-hackers(at)postgreSQL(dot)org
Subject: WIP patch for avoiding duplicate initdb runs during "make check"
Date: 2017-07-02 17:33:39
Message-ID: 19567.1499016819@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Yesterday I spent a bit of time on an idea that we've talked about
before, which is to not run initdb over and over again in contexts like
"make check-world", or even just during "make check" in contrib or in the
recovery or subscription tests. The idea would be to do it once and
then copy the created data directory tree into place for each test
run. Thus we might trade this:

$ time initdb -D $PGDATA -N
...
real 0m1.235s
user 0m0.896s
sys 0m0.341s

for this:

$ time cp -a $PGDATA junk
real 0m0.146s
user 0m0.008s
sys 0m0.136s

The attached patch is far from being committable, but it's complete enough
to get an idea of what sort of overall performance gain we might expect.
On my workstation, I've lately been doing check-world like this:
time make -s check-world -j4 PROVE_FLAGS='-j4'
and what I see is that HEAD takes about this long:
real 2m1.514s
user 2m8.113s
sys 1m15.993s
and with this patch I get
real 1m42.549s
user 0m42.888s
sys 0m46.982s
(These are median-of-3-runs; the real time bounces around a fair bit,
the CPU times not very much.)

On my laptop, a simple non-parallelized "time make check-world" takes
real 9m12.813s
user 2m15.113s
sys 1m25.631s
and with patch
real 8m3.785s
user 1m19.024s
sys 1m13.707s

So those are respectable gains, but not earth-shattering. I'm not
sure if it's worth putting more work into it right now. The main
thing that's needed to make it committable is to get rid of the
system("cp -a ...") call in favor of something more portable.
I looked longingly at our existing copydir() function but it seems
pretty tied to the backend environment. It wouldn't be hard to
make a version for frontend, but I'm not going to expend that work
right now unless people are excited enough to want to commit this
now rather than later (or not at all).

(Other unfinished work: teaching the MSVC scripts to use this,
and teaching pg_upgrade's test script to use it.)

regards, tom lane

Attachment Content-Type Size
avoid-duplicate-initdb-calls-0.1.patch text/x-diff 5.0 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tom Lane 2017-07-02 17:55:29 Re: Race-like failure in recovery/t/009_twophase.pl
Previous Message Alvaro Herrera 2017-07-02 17:32:43 Re: Race-like failure in recovery/t/009_twophase.pl