Re: warning message in standby

On Tue, Jun 29, 2010 at 7:59 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Tue, Jun 29, 2010 at 3:55 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> On Tue, Jun 15, 2010 at 11:35 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>> On the other hand, I like immediate-panicking. And I don't want the standby
>>> to retry reconnecting the master infinitely.
>>
>> On second thought, the peremptory PANIC is not good for HA system. If the
>> master unfortunately has written an invalid record because of its crash,
>> the standby would exit with PANIC before performing a failover.
>
> I don't think that should ever happen.  The master only streams WAL
> that it has fsync'd.  Presumably there's no reason for the master to
> ever fsync a partial WAL record (which is usually how a corrupt record
> gets into the stream).

This is true. But what I'm concerned about is:

1. Backend writes and fsyncs the WAL to the disk
2. The WAL on the disk gets corrupted
3. Walsender reads and sends that corrupted WAL image
4. The master crashes because of the corruption of the disk
5. The standby attempts to replay the corrupted WAL... PANIC

>> So when an invalid record is found in streamed WAL file, we should keep
>> the standby running and leave the decision whether the standby retries to
>> connect to the master forever or shuts down right now, up to the user
>> (actually, it may be a clusterware)?
>
> Well, if we want to leave it up to the user/clusterware, the current
> code is possibly adequate, although there are many different log
> messages that could signal this situation, so coding it up might not
> be too trivial.

So the current code + user-settable-retry-count seems good to me.
If the retry-count is set to 0, we will not see the repeated log
messages. And we might need to provide the parameter specifying
how the standby should behave after exceeding the retry-count:
PANIC or stay-alive-without-retries.

Choosing PANIC and using the retry-count = 5 would cover your proposed
patch.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center