Re: Re: xReader, double-effort (was: Temporary tables under hot standby)

Hello Andres,

>> The xReader design was based on the idea that it would be nice not
>> to cause load on the master machine, and that by proxying the WAL
>> stream to the HS, using synchronous replication style to write from
>> xReader to the HS, you could use the HS for a source for that data
>> with it being at exactly the right point in time to query it.
>Yes, that does make sense for some workloads. I don't think its viable for
>everything though, thats why were not aiming for that ourselves atm.

Regarding the above, what would be a case where querying the HS will not
suffice?

On Sat, Apr 28, 2012 at 4:02 AM, Andres Freund <andres(at)2ndquadrant(dot)com>wrote:

> Hi Kevin, Hi Aakash,
>
> On Saturday, April 28, 2012 12:18:38 AM Kevin Grittner wrote:
> > Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > > In the current, prototypal, state there is one component thats
> > > integrated into the server (because it needs information thats
> > > only available there).
> > The xReader design was based on the idea that it would be nice not
> > to cause load on the master machine, and that by proxying the WAL
> > stream to the HS, using synchronous replication style to write from
> > xReader to the HS, you could use the HS for a source for that data
> > with it being at exactly the right point in time to query it.
> Yes, that does make sense for some workloads. I don't think its viable for
> everything though, thats why were not aiming for that ourselves atm.
>
> > I'm not convinced that I would rather see the logic fixed inside the
> > master as opposed to being deployable on the master's machine, the
> > slave machine, or even on its own machine in between.
> I don't think that you can do everything apart from the master. We
> currently
> need shared memory for coordination between the moving parts, thats why we
> have it inside the master.
> It also have the advantage of being easier to setup.
>
> > > That component is layered ontop of a totally generic xlog
> > > reading/parsing library that doesn't care at all where its
> > > running.
> > That's cool.
>
> > > Its also used in another cluster to read the received (filtered)
> > > stream.
> > I don't quite follow what you're saying there.
> To interpret the xlog back into something that can be used for replication
> you
> need to read it again. After filtering we again write valid WAL, so we can
> use
> the same library on the sending|filtering side and on the receiving side.
> But thats actually off topic for this thread ;)
>
>
> > > I took a *very* short glance over the current wiki description of
> > > xReader and from that it seems to me it would benefit from trying
> > > to make it architecturally more similar to the rest of pg.
> > We're planning on using existing protocol to talk between pieces.
> > Other than breaking it out so that it can run somewhere other than
> > inside the server, and allowing clients to connect to xReader to
> > listen to WAL events of interest, are you referring to anything
> > else?
> It sounds like the xReader is designed to be one multiplexing process.
> While
> this definitely has some advantages resource-usage-wise it doesn't seem to
> be
> fitting the rest of the design that well. The advantages might outweigh
> everything else, but I am not sure about that.
> Something like registering/deregistering also doesn't fit that well with
> the
> way walsender works as far as I understand it.
>
> Greetings,
>
> Andres
> --
> Andres Freund http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
>