Re: Hot Backup with rsync fails at pg_clog if under load

On Sep23, 2011, at 10:41 , Heikki Linnakangas wrote:
> On 23.09.2011 11:02, Linas Virbalas wrote:
>> On 9/22/11 6:59 PM, "Euler Taveira de Oliveira"<euler(at)timbira(dot)com> wrote:
>>
>>>> If needed, I could do that, if I had the exact procedure... Currently,
>>>> during the start of the backup I take the following information:
>>>>
>>> Just show us the output of pg_start_backup and part of the standby log with
>>> the following message 'redo starts at' and the subsequent messages up to the
>>> failure.
>>
>> Unfortunately, it's impossible, because the error message "Could not read
>> from file "pg_clog/0001" at offset 32768: Success" is shown (and startup
>> aborted) before the turn for "redo starts at" message arrives.
>
> It looks to me that pg_clog/0001 exists, but it shorter than recovery expects. Which shouldn't happen, of course, because the start-backup checkpoint should flush all the clog that's needed by recovery to disk before the backup procedure begins to them.

Yeah. What confuses me though is that we fail while trying to read from the clog. When do we do that during normal (non-standby) recovery?

One other possibility would be that the problem is somehow triggered by vacuum running while the start-backup checkpoint is commencing. That'd explain why the problem goes away with immediate checkpoints, because those make the windows mach smaller. But I haven't got a concrete theory of whats happening..

best regards,
Florian Pflug