Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> I have observed one more design issue.
>

Good observation.

> The problem is that when we
> get a toasted chunks we remember the changes in the memory(hash table)
> but don't stream until we get the actual change on the main table.
> Now, the problem is that we might get the change of the toasted table
> and the main table in different streams. So basically, in a stream,
> if we have only got the toasted tuples then even after
> ReorderBufferStreamTXN the memory usage will not be reduced.
>

I think we can't split such changes in a different stream (unless we
design an entirely new solution to send partial changes of toast
data), so we need to send them together. We can keep a flag like
data_complete in ReorderBufferTxn and mark it complete only when we
are able to assemble the entire tuple. Now, whenever, we try to
stream the changes once we reach the memory threshold, we can check
whether the data_complete flag is true, if so, then only send the
changes, otherwise, we can pick the next largest transaction. I think
we can retry it for few times and if we get the incomplete data for
multiple transactions, then we can decide to spill the transaction or
maybe we can directly spill the first largest transaction which has
incomplete data.

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com