Re: How can we make beta testing better?

On Thu, Apr 17, 2014 at 5:26 PM, Jeff Janes <jeff(dot)janes(at)gmail(dot)com> wrote:
> A lot of the bugs that turned up are not the kind I would expect to have
> been found in most beta testing done by non-hacking users. Weren't they
> mostly around rare race conditions, crash recovery, and freezing?

Actually I was struck by how the bugs in 9.3 were the kinds of bugs
that should have turned up pretty quickly by user testing. They
certainly turned up pretty quickly after users put their production
applications on it. They *didn't* require rare race conditions, just
certain patterns of workloads for long enough to reliably reproduce.

They were specifically *not* the kinds of bugs that regression testing
would have found. Regression testing only finds bugs you anticipate
and think to put in the specification of correct behaviour. If you had
thought of these problems you would have tested them manually and in
any case you would have seen the omissions immediately on inspected
the code.

Crash recovery and freezing aren't rare things once you have hot
standbys everywhere and run 24x7 applications (or load tests) on your
systems. We could make freezing more frequent by having a mode that
bumps the xid by a few million randomly. That would still be pretty
hit and miss whether it happens to wrap around in any particular
state.

--
greg