Re: slowest tap tests - split or accelerate?

On Mon, Jan 17, 2022 at 2:57 PM Andres Freund <andres(at)anarazel(dot)de> wrote:
> > pg_basebackup's 010_pg_basebackup.pl looks like it could be split up,
> > though. That one, at least to me, looks like people have just kept
> > adding semi-related things into the same test file.
>
> Yea.

Here's a patch that splits up that file. Essentially the first half of
the file is concerned with testing that a backup ends up in the state
it expects, while the second half is concerned with checking that
various options to pg_basebackup work. So I split it that way, plus I
moved some of the really basic stuff to a completely separate file
with a very brief runtime. The test results are interesting.

Unpatched:

[12:33:33] t/010_pg_basebackup.pl ... ok 16161 ms ( 0.02 usr 0.00
sys + 2.07 cusr 7.80 csys = 9.89 CPU)
[12:33:49] t/020_pg_receivewal.pl ... ok 4115 ms ( 0.00 usr 0.00
sys + 0.89 cusr 1.73 csys = 2.62 CPU)
[12:33:53] t/030_pg_recvlogical.pl .. ok 1857 ms ( 0.01 usr 0.01
sys + 0.63 cusr 0.73 csys = 1.38 CPU)
[12:33:55]
All tests successful.
Files=3, Tests=177, 22 wallclock secs ( 0.04 usr 0.02 sys + 3.59
cusr 10.26 csys = 13.91 CPU)

Pached:

[12:32:03] t/010_pg_basebackup_basic.pl ...... ok 192 ms ( 0.01
usr 0.00 sys + 0.10 cusr 0.05 csys = 0.16 CPU)
[12:32:03] t/011_pg_basebackup_integrity.pl .. ok 5530 ms ( 0.00
usr 0.00 sys + 0.87 cusr 2.51 csys = 3.38 CPU)
[12:32:09] t/012_pg_basebackup_options.pl .... ok 13117 ms ( 0.00
usr 0.00 sys + 1.87 cusr 6.31 csys = 8.18 CPU)
[12:32:22] t/020_pg_receivewal.pl ............ ok 4314 ms ( 0.01
usr 0.00 sys + 0.97 cusr 1.77 csys = 2.75 CPU)
[12:32:26] t/030_pg_recvlogical.pl ........... ok 1908 ms ( 0.00
usr 0.00 sys + 0.64 cusr 0.77 csys = 1.41 CPU)
[12:32:28]
All tests successful.
Files=5, Tests=177, 25 wallclock secs ( 0.04 usr 0.02 sys + 4.45
cusr 11.41 csys = 15.92 CPU)

Sadly, we've gained about 2.5 seconds of runtime as the price of
splitting the test. Arguably the options part could be split up a lot
more finely than this, but that would drive up the runtime even more,
basically because we'd need more initdbs. So I don't know whether it's
better to leave things as they are, split them this much, or split
them more. I think this amount of splitting might be justified simply
in the interests of clarity, but I'm reluctant to go further unless we
get some nifty initdb-caching system.

Thoughts?

--
Robert Haas
EDB: http://www.enterprisedb.com