Re: Improve error detections in TAP tests by spreading safe_psql

On Wed, Aug 28, 2019 at 09:44:58PM -0400, Tom Lane wrote:
> Well, it's useful if you just want the stdout back. But its name
> is a bit misleading if the default behavior of psql is just as
> safe. Not sure whether renaming it is worthwhile.

It is not that complicated enough to capture stdout with
PostgresNode::psql either, so if we are making the default of one
(PostgresNode::psql) look like the other (PostgresNode::safe_psql),
then we lose no actual feature by dropping safe_psql.

> Yeah, but only if the test cases are independent, which I think is
> mostly not true in our TAP scripts. Otherwise you're just looking
> at cascading errors.

Yep. We have a couple of cases though where things are quite
independent, like the 2PC suite divided into sequences of different
transactions, but mostly there are dependencies between one step and
the follow-up ones.

> Yup, the tradeoff is people possibly having test scripts outside
> our tree that would break, versus the possibility that we'll back-patch
> test changes that don't behave as expected in back branches. I don't
> know if the former is true, but I'm afraid the latter is a certainty
> if we don't back-patch.

Exactly, that's why I am wondering how wide breakages in those
out-of-the-tree tests would go as we have PostgresNode since 9.6.
Facing this choice makes me uneasy, which is why I would tend to only
do incompatible things on HEAD. Another, safer, possibility would be
to introduce in back-branches an extra psql-like routine enforcing
errors which we use in our tests, to keep the former ones for
compatibility, and to drop the old ones on HEAD. It is never fun to
face sudden errors on a system when doing a minor upgrade.
--
Michael