Re: pgsql: Remove pgbench "progress" test pending solution of its timing is

Hello Tom,

> Well, I think it's mostly about valgrind making everything really slow.
> Since we have seen some passes from skink recently, perhaps there was
> also a component of more-load-on-the-machine-than-usual. But in the end
> this is just evidence for my point that regression tests have to be very
> much not timing-sensitive. We run them under all kinds of
> out-of-the-ordinary stress.

Attached is an attempt at improving the situation:

(1) there are added comments to explain the whys, which is to provide
coverage for pgbench time-related features... while still not being
time-sensitive, which is a challenge.

(2) the test now only expects "progress: \d" from the output, so it is
enough that one progress is shown, whenever it is shown.

(3) if the test is detected to have gone AWOL, detailed log checks are
coldly skipped.

This would have passed on "skink" under the special conditions it
encountered.

I cannot guaranty that it would pass under any circumstance, though.

If it still encounters a failure, ISTM that it should only be a missing
"progress:" in the output, which has not been encountered so far.

If it occurs, a few options would remain, none of them very convincing:

- give the test some more time, eg 3 seconds (hmmm... could still fail
after any time...)

- drop the "progress:" expectation (hmmm... but then it does not check
anything).

- make the "progress:" output check conditional to the running time
(hmmm... it would require changing the command_checks_all function,
and there is still a chance that the bench was stuck for 2 seconds
then given time to realize that it has to stop right now...)

- use an even simpler transaction, eg "SELECT;" which is less likely to
get stuck (but still could get stuck...).

For the random-ness related test (--sample-rate), we could add a feature
to pgbench which allows to control the random seed, so that the number of
samples could be known in advance for testing purposes.

--
Fabien.