Re: Assertion failure while streaming toasted data

On Tue, May 25, 2021 at 2:57 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:

>
> >
> > Yes, I am able to reproduce this, basically, until we get the last
> > tuple of the multi insert we can not clear the toast data otherwise we
> > can never form a complete tuple. So the only possible fix I can think
> > of is to consider the multi-insert WAL without the final multi-insert
> > tuple as partial data then we will avoid streaming until we get the
> > complete WAL of one multi-insert. I am working on the patch to fix
> > this, I will share that in some time.
>
> The attached patch should fix the issue, now the output is like below
>
>
Thanks. This looks fine to me. We should still be able to stream
multi-insert transactions (COPY) as and when the copy buffer becomes full
and is flushed. That seems to be a reasonable restriction to me.

We should incorporate the regression test in the final patch. I am not
entirely sure if what I have done is acceptable (or even works in
all scenarios). We could possibly have a long list of tuples instead of
doing the exponential magic. Or we should consider lowering the min value
for logical_decoding_work_mem and run these tests with a much lower value.
In fact, that's how I caught the problem in the first place. I had
deliberately lowered the value to 1kB so that streaming code kicks in very
often and even for small transactions.

Thanks,
Pavan

--
Pavan Deolasee
EnterpriseDB: https://www.enterprisedb..com