Skip site navigation (1) Skip section navigation (2)

Re: Recovery bug

From: Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com>
To: Jeff Davis <pgsql(at)j-davis(dot)com>
Cc: pgsql-bugs(at)postgresql(dot)org
Subject: Re: Recovery bug
Date: 2010-10-19 09:26:48
Message-ID: 4CBD6458.4050400@enterprisedb.com (view raw or flat)
Thread:
Lists: pgsql-bugs
On 18.10.2010 01:48, Jeff Davis wrote:
> On Fri, 2010-10-15 at 15:58 -0700, Jeff Davis wrote:
>> I don't have a fix yet, because I think it requires a little discussion.
>> For instance, it seems to be dangerous to assume that we're starting up
>> from a backup with access to the archive when it might have been a crash
>> of the primary system. This is obviously wrong in the case of an
>> automatic restart, or one with no restore_command. Fixing this issue
>> might also remove the annoying "If you are not restoring from a backup,
>> try removing..." PANIC error message.
>>
>> Also, in general we should do more logging during recovery, at least the
>> first stages, indicating what WAL segments it's looking for to get
>> started, why it thinks it needs that segment (from backup or control
>> data), etc. Ideally we would verify that the necessary files exist (at
>> least the initial ones) before making permanent changes. It was pretty
>> painful trying to work backwards on this problem from the final
>> controldata (where checkpoint and prior checkpoint are the same, and
>> redo is before both), a crash, a PANIC, a backup_label.old, and not much
>> else.
>>
>
> Here's a proposed fix. I didn't solve the problem of determining whether
> we really are restoring a backup, or if there's just a backup_label file
> left around.
>
> I did two things:
>    1. If reading a checkpoint from the backup_label location, verify that
> the REDO location for that checkpoint exists in addition to the
> checkpoint itself. If not, elog with a FATAL immediately.

Makes sense. I wonder if we could just move the rename() after reading 
the checkpoint record?

>    2. Change the error that happens when the checkpoint location
> referenced in the backup_label doesn't exist to a FATAL. If it can
> happen due to a normal crash, a FATAL seems more appropriate than a
> PANIC.

I guess, although it's really not appropriate that the database doesn't 
recover after a crash during a base backup.

> I still think it would be nice if postgres knew whether it was restoring
> a backup or recovering from a crash, otherwise it's hard to
> automatically recover from failures. I thought about using the presence
> of recoveryRestoreCommand or PrimaryConnInfo to determine that. But it
> seemed potentially dangerous if the person restoring a backup simply
> forgot to set those, and then it tries restoring from the controldata
> instead (which is unsafe to do during a backup).

Right, that's not good either.

One alternative is to not remove any WAL files during a base backup. The 
obvious downside is that if the backup takes a long time, you run out of 
disk space.

The fundamental problem is that by definition, a base backup is 
completely indistinguishable from the data directory in the original 
server. Or is it? We recommend that you exclude the files under pg_xlog 
from the backup. So we could create a "pg_xlog/just_kidding" file along 
with backup_label. When starting recovery, if just_kidding exists, we 
can assume that we're doing crash recovery and ignore backup_label.

Excluding pg_xlog is just a recommendation at the moment, though, so we 
would need a big warning in the docs. And some way to enforce that 
just_kidding is not included in the backup would be nice, maybe we could 
remove read-permission from it?

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com

In response to

Responses

pgsql-bugs by date

Next:From: Richard HuxtonDate: 2010-10-19 13:35:01
Subject: BUG #5717: Domain as array of numeric/varchar does not respect limits
Previous:From: Fujii MasaoDate: 2010-10-19 08:31:07
Subject: Re: Recovery bug

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group