Re: Index corruption with CREATE INDEX CONCURRENTLY

On Sun, Feb 5, 2017 at 9:42 PM, Pavan Deolasee <pavan(dot)deolasee(at)gmail(dot)com> wrote:
> On Mon, Feb 6, 2017 at 5:41 AM, Peter Geoghegan <pg(at)bowt(dot)ie> wrote:
>> On Sun, Feb 5, 2017 at 4:09 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> > I don't think this kind of black-and-white thinking is very helpful.
>> > Obviously, data corruption is bad. However, this bug has (from what
>> > one can tell from this thread) been with us for over a decade; it must
>> > necessarily be either low-probability or low-severity, or somebody
>> > would've found it and fixed it before now. Indeed, the discovery of
>> > this bug was driven by new feature development, not a user report. It
>> > seems pretty clear that if we try to patch this and get it wrong, the
>> > effects of our mistake could easily be a lot more serious than the
>> > original bug.
>>
>> +1. The fact that it wasn't driven by a user report convinces me that
>> this is the way to go.
>>
>
> I'm not sure that just because the bug wasn't reported by a user, makes it
> any less critical. As Tomas pointed down thread, the nature of the bug is
> such that the users may not discover it very easily, but that doesn't mean
> it couldn't be affecting them all the time. We can now correlate many past
> reports of index corruption to this bug, but we don't have evidence to prove
> that. Lack of any good tool or built-in checks probably makes it even
> harder.
>
> The reason why I discovered this with WARM is because it now has a built-in
> recheck logic, which was discarding some tuples returned by the index scan.
> It took me lots of code review and inspection to finally conclude that this
> must be an existing bug. Even then for lack of any utility, I could not
> detect this easily with master. That doesn't mean I wasn't able to
> reproduce, but detection was a problem.
>
> If you look at the reproduction setup, one in every 10, if not 5, CREATE
> INDEX CONCURRENTLY ends up creating a corrupt index. That's a good 10%
> probability. And this is on a low end laptop, with just 5-10 concurrent
> clients running. Probability of hitting the problem will be much higher on a
> bigger machine, with many users (on a decent AWS machine, I would find every
> third CIC to be corrupt). Moreover the setup is not doing anything
> extraordinary. Just concurrent updates which change between HOT to non-HOT
> and a CIC.
>

Not that I am advocating that we should do a release just for this;
having a fix we believe in is certainly as important a factor, but
that the idea that the bug has been around a long time means it is
less of an issue does seem wrong. We've certainly seen plenty of cases
over the years where bugs have existed in the code in seldom used code
paths, only to be exposed by new features or other code changes over
time. In general, we should be less worried about the age of a bug vs
our expectations that users are likely to hit that bug now, which does
seem high based on the above numbers.

In the meantime, it's certainly worth warning users, providing help on
how to determine if this is a likely problem for them, and possibly
rolling a patch ahead of upstream in cases where that's feasible.

Robert Treat
play: xzilla.net
work: omniti.com