Allow cancelling VACUUM of nbtrees with corrupted right links

From: Andres Freund <andres(at)anarazel(dot)de>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Allow cancelling VACUUM of nbtrees with corrupted right links
Date: 2018-06-27 19:16:29
Message-ID: 20180627191629.wkunw2qbibnvlz53@alap3.anarazel.de
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

Hi,

A couple times, one very recently, I've encountered btrees that somehow
had corrupted right links. The links formed a cycle, involving a number
of pages. As of yet it's unclear to me where the corruption is
originating from - could be a storage issue or a postgres issue.

What makes that kind of corruption annoying is not so much lookups or
insertsion not working, but that it can lead to VACUUM being stuck. Page
deletion codepaths have to follow right links, and if there's a cycle
they'll do so forever. That'd be bad enough, but there's no
CHECK_FOR_INTERRUPTS() in those codepaths, which means autovacuum can't
be cancelled. And thus the index can't easily be dropped / reindexed.

In an older case that lead to significant difficulty for the user to
ever get out of the situation, because even after a shutdown autovacuum
quickly latched onto the table, IIRC due to an impeding wraparound.

I think it'd be a good minimal fix if we added a bunch of CFIs to the
likely instances of such loops. I've done that in the attached patch.
Unfortunately it's entirely trivial, because CFI will not trigger when
an lwlock is held (as LWLockAcquire() does a HOLD_INTERRUPTS()). Any
comments about the patch?

I couldn't see how to fix the _bt_unlink_halfdead_page() right-sib loop,
because we always hold a lock. But given that that loop appears to
mostly be dead code, that doesn't seem too bad?

I think we should backpatch those checks - it's a fairly nasty situation
for users to not be able to even drop an index without going to single
user mode.

I wonder if we additionally should put a CFI() in _bt_relandgetbuf(), as
it's otherwise impossible to check interrupts at the callsites.

Alternatively we could also invent a CFI version that allows
cancellation even if locks held - but that seems nontrivial.

Greetings,

Andres Freund

Attachment Content-Type Size
0001-Check-for-interrupts-inside-nbtree-page-deletion-cod.patch text/x-diff 2.2 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Alvaro Herrera 2018-06-27 19:18:19 Re: Remove mention in docs that foreign keys on partitioned tables are not supported
Previous Message Daniel Gustafsson 2018-06-27 18:57:43 Re: [HACKERS] Support for Secure Transport SSL library on macOS as OpenSSL alternative