Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

From: Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com>
To: Erikjan Rijkers <er(at)xs4all(dot)nl>
Cc: PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
Date: 2017-12-23 20:06:05
Message-ID: 1cc7daa2-027e-d490-2396-96e16d328c45@2ndquadrant.com
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

On 12/23/2017 03:03 PM, Erikjan Rijkers wrote:
> On 2017-12-23 05:57, Tomas Vondra wrote:
>> Hi all,
>>
>> Attached is a patch series that implements two features to the logical
>> replication - ability to define a memory limit for the reorderbuffer
>> (responsible for building the decoded transactions), and ability to
>> stream large in-progress transactions (exceeding the memory limit).
>>
>
> logical replication of 2 instances is OK but 3 and up fail with:
>
> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
> "reorderbuffer.c", Line: 1773)
>
> I can cobble up a script but I hope you have enough from the assertion
> to see what's going wrong...

The assertion says that the iterator produces changes in order that does
not correlate with LSN. But I have a hard time understanding how that
could happen, particularly because according to the line number this
happens in ReorderBufferCommit(), i.e. the current (non-streaming) case.

So instructions to reproduce the issue would be very helpful.

Attached is v2 of the patch series, fixing two bugs I discovered today.
I don't think any of these is related to your issue, though.

regards

--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachment Content-Type Size
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch.gz application/gzip 6.8 KB
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch.gz application/gzip 2.7 KB
0003-Issue-individual-invalidations-with-wal_level-log-v2.patch.gz application/gzip 4.8 KB
0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch.gz application/gzip 5.3 KB
0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch.gz application/gzip 10.6 KB
0006-Add-support-for-streaming-to-built-in-replication-v2.patch.gz application/gzip 13.9 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Maksim Milyutin 2017-12-23 20:23:43 Re: PoC: custom signal handler for extensions
Previous Message Erikjan Rijkers 2017-12-23 14:03:02 Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions