Re: Multi-xacts and our process problem

On Tue, May 12, 2015 at 02:42:16AM -0400, Noah Misch wrote:
> > > The "revert or rework" ship had already sailed at that point. I
> >
> > True.
> >
> > > don't think we had much choice than just soldier through the bugs
> > > after the release.
> >
> > The problem is we "soldiered on" without adding any resources to the
> > problem or doing a systematic review once it became clear one was
> > necessary.
>
> In both this latest emergency and the Nov-Dec 2013 one, several people showed
> up and helped. We did well in that respect, but your idea that we should have
> started a systematic post-commit review is a good one. Hoping other parts of
> the change were fine, the first team dispersed.

Yes, Alvaro faithfully addressed each bug that was reported and applied
a fix, and then we waited for the next reported bug, which happened
repeatedly. What didn't happen is someone realizing that we had a
situation that required a _different_ approach. We just stuck to the
approach that had always worked in the past and never revisited it.

Some people think we need a better plan, but a larger issue is that we
need to recognize when our existing plan _isn't_ working and do
something different. No matter what plan or procedure we create, there
are going to be cases where it doesn't work, and if everyone is too busy
to recognize that our plan isn't working, these mistakes will be
repeated. Our existing plan had worked so well for so many years that
it took very long for us to recognize we needed a new approach in this
case.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +