Re: pg_basebackup blocking all queries with horrible performance

From: Lonni J Friedman <netllama(at)gmail(dot)com>
To: Jerry Sievers <gsievers19(at)comcast(dot)net>
Cc: Magnus Hagander <magnus(at)hagander(dot)net>, pgsql-admin(at)postgresql(dot)org
Subject: Re: pg_basebackup blocking all queries with horrible performance
Date: 2012-06-08 01:01:46
Message-ID: CAP=oouF_gTLJvxQ8tK8czZ_wRF3Xt63j40LJnBfbHE26qfOYgg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin pgsql-hackers

On Thu, Jun 7, 2012 at 5:07 PM, Jerry Sievers <gsievers19(at)comcast(dot)net> wrote:
> Lonni J Friedman <netllama(at)gmail(dot)com> writes:
>
>> On Thu, Jun 7, 2012 at 12:40 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>
>>> On Thu, Jun 7, 2012 at 8:04 PM, Lonni J Friedman <netllama(at)gmail(dot)com> wrote:
>>>> On Thu, Jun 7, 2012 at 10:41 AM, Lonni J Friedman <netllama(at)gmail(dot)com> wrote:
>>>>> Greetings,
>>>>> I have a 4 server postgresql-9.1.3 cluster (one master doing streaming
>>>>> replication to 3 hot standby servers).  All of them are running
>>>>> Fedora-16-x86_64.
>>>>>
>>>>> http://wiki.postgresql.org/wiki/Lock_Monitoring
>>>>
>>>> err, i included that URL but neglected to explain why.  On a different
>>>> list someone suggested that I verify that there were no locks that
>>>> were blocking things, and I did so, and found no locks.
>>>>
>>>> So I'm still at a loss why pg_basebackup is killing perf, and would
>>>> appreciate pointers on how to debug it or at least reduce its impact
>>>> on performance if that is possible.
>>>>
>>>
>>> My guess would be that you are overloading your I/O system. You should
>>> look at values from iostat and vmstat from when the system works fine
>>> and when you run pg_basebackup, that should give you a hint in the
>>> right direction.
>>
>> ok, thanks.  i'll take a look at that.  If this turns out to be the
>> issue, is there some way to get pg_basebackup to run more slowly, so
>> that it has less impact?  Or could I do this with ionice on the
>> pg_basebackup process?
>
> You might try stopping pg_basebackup in place with SIGSTOP and check
> if problem goes away.  SIGCONT and you should  start having
> sluggishness again.
>
> If verified, then any sort of throttling mechanism should work.

I'm certain that the problem is triggered only when pg_basebackup is
running. Its very predictable, and goes away as soon as pg_basebackup
finishes running. What do you mean by a throttling mechanism?

In response to

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Craig Ringer 2012-06-08 05:56:38 (new thread) could not rename temporary statistics file "pg_stat_tmp/pgstat.tmp" to "pg_stat_tmp/pgstat.stat": No such file or directory
Previous Message Jerry Sievers 2012-06-08 00:07:24 Re: pg_basebackup blocking all queries with horrible performance

Browse pgsql-hackers by date

  From Date Subject
Next Message Simon Riggs 2012-06-08 01:25:16 Re: Avoiding adjacent checkpoint records
Previous Message Sergey Koposov 2012-06-08 00:28:15 Re: slow dropping of tables, DropRelFileNodeBuffers, tas