Re: pg_regress: Treat child process failure as test failure

Hi,

On 2022-11-26 21:11:39 +0100, Daniel Gustafsson wrote:
> In the thread about TAP format out in pg_regress, Andres pointed out [0] that
> we allow a test to pass even if the test child process failed. While its
> probably pretty rare to have a test pass if the process failed, this brings a
> risk for false positives (and it seems questionable that any regress test will
> have a child process failing as part of its intended run).

> The attached makes child failures an error condition for the test as a belts
> and suspenders type check. Thoughts?

I wonder if it's the right thing to treat a failed psql that's then also
ignored as "failed (ignored)". Perhaps it'd be better to move the statuses[i]
!= 0 check to before the if (differ)?

> - if (differ)
> + if (differ || statuses[i] != 0)
> {
> bool ignore = false;
> _stringlist *sl;
> @@ -1815,7 +1815,7 @@ run_single_test(const char *test, test_start_function startfunc,
> differ |= newdiff;
> }
>
> - if (differ)
> + if (differ || exit_status != 0)
> {
> status(_("FAILED"));
> fail_count++;

It certainly is a bit confusing that we print a psql failure separately from
the if "FAILED" vs "ok" bit.

Greetings,

Andres Freund