Re: Unarchived WALs deleted after crash

On 15 February 2013 14:31, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> wrote:
> On 14.02.2013 17:45, Jehan-Guillaume de Rorthais wrote:
>>
>> I am facing an unexpected behavior on a 9.2.2 cluster that I can
>> reproduce on current HEAD.
>>
>> On a cluster with archive enabled but failing, after a crash of
>> postmaster, the checkpoint occurring before leaving the recovery mode
>> deletes any additional WALs, even those waiting to be archived.
>
>> ...
>> Is it expected ?
>
> No, it's a bug. Ouch. It was introduced in 9.2, by commit
> 5286105800c7d5902f98f32e11b209c471c0c69c:

Thanks for tracking that down.

>> - /*
>> - * Normally we don't delete old XLOG files during recovery to
>> - * avoid accidentally deleting a file that looks stale due to a
>> - * bug or hardware issue, but in fact contains important data.
>> - * During streaming recovery, however, we will eventually fill the
>> - * disk if we never clean up, so we have to. That's not an issue
>> - * with file-based archive recovery because in that case we
>> - * restore one XLOG file at a time, on-demand, and with a
>> - * different filename that can't be confused with regular XLOG
>> - * files.
>> - */
>> - if (WalRcvInProgress() || XLogArchiveCheckDone(xlde->d_name))
>> + if (RecoveryInProgress() || XLogArchiveCheckDone(xlde->d_name))
>> [ delete the file ]
>
>
> With that commit, we started to keep WAL segments restored from the archive
> in pg_xlog, so we needed to start deleting old segments during archive
> recovery, even when streaming replication was not active. But the above
> change was to broad; we started to delete old segments also during crash
> recovery.
>
> The above should check InArchiveRecovery, ie. only delete old files when in
> archive recovery, not when in crash recovery. But there's one little
> complication: InArchiveRecovery is currently only valid in the startup
> process, so we'll need to also share it in shared memory, so that the
> checkpointer process can access it.
>
> I propose the attached patch to fix it.

Agree with your diagnosis and fix.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services