Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Mon, Mar 30, 2020 at 8:58 PM Tomas Vondra
<tomas(dot)vondra(at)2ndquadrant(dot)com> wrote:
>
> On Mon, Mar 30, 2020 at 11:47:57AM +0530, Amit Kapila wrote:
> >
> >I think we need to cache the subxids on the replica somehow but I
> >don't have a very good idea for it. Basically, there are two ways to
> >do it (a) Change the KnownAssignedXids in some way so that we can
> >easily find this information without losing on the current benefits of
> >it. I can't think of a good way to do that and even if we come up
> >with something, it could easily be a lot of work, (b) Cache the
> >subxids for a particular transaction in local memory along with
> >KnownAssignedXids. This is doable but now we have two data-structures
> >(one in shared memory and other in local memory) managing the same
> >information in different ways.
> >
> >Do you have any other ideas?
>
> I don't follow. Why couldn't we have a simple cache on the standby? It
> could be either a simple array or a hash table (with the top-level xid
> as hash key)?
>

I think having something like we discussed or what you have in the
patch won't be sufficient to clean the KnownAssignedXid array. The
point is that we won't write a WAL for xid-subxid association for
unlogged relations in the "Immediately WAL-log assignments" patch,
however, the KnownAssignedXid would have both kinds of Xids as we
autofill it with gaps (see RecordKnownAssignedTransactionIds). I
think if my understanding is correct to make it work we might need
major surgery in the code or have to maintain KnownAssignedXid array
differently.

>
> I don't think this is particularly complicated or a lot of code, and I
> don't see why would it require data structures in shared memory. Only
> the walreceiver on standby needs to worry about this, no?
>

Not a new data structure in shared memory, but we already have a
KnownTransactionId structure in shared memory. So, after having a
local cache, we will have xidAssignmentsHash and KnownTransactionId
maintaining the same information in different ways. And, we need to
ensure both are cleaned up properly. That was what I was pointing
above related to maintaining two structures. However, I think before
discussing more on this, we need to think about the above problem.

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com