Re: tests and meson - test names and file locations

Hi,

On 2022-08-12 18:08:00 +0200, Peter Eisentraut wrote:
> > At the moment there's three suites differentiating by the type of test:
> > 'pg_regress', 'isolation' and 'tap'. There's also a separate "axis" of suites,
> > describing what's being tested, e.g. 'main', 'test_decoding', 'recovery' etc.
> >
> > That currently works out to each test having two suites, although I've
> > wondered about adding a 'contrib' suite as well.
>
> I'm not sure what the value of these suite names would be. I don't usually
> find myself wanting to run, say, just all tap tests.

I occasionally want to, but it may not be important enough. I do find it
useful for display purposes alone, and for finding kinds of test with meson
test --list.

> On 11.08.22 06:04, Andres Freund wrote:
> > = Log and Data locations =
> >
> > To make things like the selection of log files for a specific test easier,
> > I've so far set it up so that test data and logs are stored in a separate
> > directory from the sources.
> >
> > testrun/<main|recovery|...>/<testname>/<log|tmp_check|results...>
> >
> > The runner now creates a test.start at the start of a test and either
> > test.success or test.failure at the end. That should make it pretty easy for
> > e.g. the buildfarm and CI to make the logs for a failed test easily
> > accessible. I've spent far too much time going through the ~hundred logs in
> > src/test/recovery/ that the buildfarm displays as one thing.
>
> I don't really understand which problem this solves and how. Sure, the test
> output is somewhat complex, but I know where it is and I've never found
> myself wishing it to be somewhere else.

I'd like the buildfarm and CI a) use parallelism to run tests (that's why the
BF is slow) b) show the logfiles for exactly the failed test ([1]). We can of
course iterate through the whole directory tree, somehow identify which log
files are for which test, and then select the log files for the failed
tests. But that's much easier to do then when you have a uniform directory
hierarchy, where you can test which tests have failed based on the filesystem
alone.

Greetings,

Andres Freund

[1] E.g. the log file for this failed run is 13MB, and I've had ones with a
bit more debugging enabled crash both firefox and chrome before
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=skink&dt=2022-08-06%2012%3A17%3A14&stg=recovery-check