Re: Testing "workers launched" in expected output? Really?

On 3 March 2018 at 04:27, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Fri, Mar 2, 2018 at 3:13 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > My point is that just because it isn't falling over on
> > relatively-lightly-loaded buildfarm machines doesn't mean that it won't
> > fall over in other environments.
>
> It doesn't mean it will, either.
>
> > I should think that your recent experience with postgres_fdw (which is
> > evidently still broken, BTW, see rhinoceros) would have convinced you
> > that environmentally dependent regression test results are something to
> > studiously avoid.
>
> I suspect the problem with the test is that it's testing for a plan
> that is only slightly better than the next-best plan, and so it takes
> very little to push it over the edge. That's sort of an environment
> issue, but it's not exactly the same thing you're worried about here.
>
> > We have enough trouble with unforeseen test deltas;
> > putting in tests that have a blatantly obvious failure mechanism is
> > just taunting the software gods. Who tend to take their revenge in
> > unpleasant ways.
> >
> > If you insist on actual failures, perhaps I'll go set up about four
> > concurrently-scheduled animals on prairiedog's host, and wait to see
> > what happens ...
>
> I will be curious to see the results of that experiment. Unless you
> press the machine so hard that fork() starts returning EAGAIN, I don't
> see why that should cause this to fail. And if you do that, then it's
> going to fail far beyond the ability of suppressing Workers Launched
> to cover the problem up. However, it's certainly possible that you've
> thought of some failure mode that I've missed, or that there is one
> which neither of us has considered.
>

We already perpetrate all sorts of horrid things to work around
pg_regress's literal whole-file expected output matching. Hard-to-maintain
and impossible-to-document alternates files being high on my list.

Rather than adding to the list of things we hide and suppress to cover the
deficiencies in our regression test results pattern matching I wonder if
it's time to fix the pattern matching. Provide for some limited wildcard
features in pg_regress output pattern files, for example.

Easier said than done without requiring a total change in how all expected
files are written, unfortunately. "magic tags" to open a regexp section of
expected file are all well and good, but then we can't just use diff
anymore...

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services