Re: Need a builtin way to run all tests faster manner

On 2017-03-07 09:36:51 -0300, Alvaro Herrera wrote:
> FWIW, +1 on improving matters here.
>
> Andres Freund wrote:
>
> > The best I can come up so far is a toplevel target that creates the temp
> > install, starts a cluster and then runs the 'installcheck-or-check'
> > target on all the subdirectories via recursion. Individual makefiles can
> > either use the pre-existing cluster (most of of contrib for example), or
> > use the temporary install and run their pre-existing check target using
> > that (the tap tests, test_decoding, ...).
>
> I think a toplevel installcheck-or-check target is a good first step
> (though definitely lets find a better name). Just being able to run all
> tests without the need for 95% of pointless initdb's would be helpful
> enough.

Here's a hacky proof-of-concept patch that implements a contrib/
'fastcheck' target.
timing of:

make -s -j01 -Otarget check 2m49.286s
make -s -j16 -Otarget check 0m44.120s
make -s -j01 -Otarget fastcheck 1m1.533s
make -s -j16 -Otarget fastcheck 0m24.385s
make -s -j01 -Otarget USE_MODULE_DB=1 \
installcheck check-test_decoding-recurse 0m56.016s
make -s -j16 -Otarget USE_MODULE_DB=1 \
installcheck check-test_decoding-recurse 0m23.608s

All of these should, excepting rerunning initdb/server start, have
equivalent test coverage.

'fastcheck' is obviously, if you look at it, a POC. We'd need:
- abstract this away somehow, it's not nice to copy the logic into a
makefile
- tie contrib/'fastcheck' into check-world by overriding that target
- use the temp install we've created previously?
- more bulletproof server stop mechanism?

- Andres