From: | Stas Kelvich <s(dot)kelvich(at)postgrespro(dot)ru> |
---|---|
To: | Petr Jelinek <petr(dot)jelinek(at)2ndquadrant(dot)com> |
Cc: | pgsql-hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: Logical replication ApplyContext bloat |
Date: | 2017-04-19 09:55:13 |
Message-ID: | E21FD441-90A6-4B57-8ED6-2911492E60B5@postgrespro.ru |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
> On 19 Apr 2017, at 12:37, Petr Jelinek <petr(dot)jelinek(at)2ndquadrant(dot)com> wrote:
>
> On 18/04/17 13:45, Stas Kelvich wrote:
>> Hi,
>>
>> currently logical replication worker uses ApplyContext to decode received data
>> and that context is never freed during transaction processing. Hence if publication
>> side is performing something like 10M row inserts in single transaction, then
>> subscription worker will occupy more than 10G of ram (and can be killed by OOM).
>>
>> Attached patch resets ApplyContext after each insert/update/delete/commit.
>> I’ve tried to reset context only on each 100/1000/10000 value of CommandCounter,
>> but didn’t spotted any measurable difference in speed.
>>
>> Problem spotted by Mikhail Shurutov.
>>
>
> Hmm we also do replication protocol handling inside the ApplyContext
> (LogicalRepApplyLoop), are you sure this change is safe in that regard?
Memory is bloated by logicalrep_read_* when palloc happens inside.
I’ve inserted context reset in ensure_transaction() which is only called in beginning
of hande_insert/update/delete/commit when data from previous call of that
function isn’t needed. So probably it is safe to clear memory there. At least
i don’t see any state that can be accessed later in this functions. Do you
have any specific concerns?
Stas Kelvich
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
From | Date | Subject | |
---|---|---|---|
Next Message | Petr Jelinek | 2017-04-19 10:24:46 | Re: Logical replication ApplyContext bloat |
Previous Message | Petr Jelinek | 2017-04-19 09:37:25 | Re: Logical replication ApplyContext bloat |