Re: Re: xReader, double-effort (was: Temporary tables under hot standby)

Hi Kevin, Hi Aakash,

On Saturday, April 28, 2012 12:18:38 AM Kevin Grittner wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > In the current, prototypal, state there is one component thats
> > integrated into the server (because it needs information thats
> > only available there).
> The xReader design was based on the idea that it would be nice not
> to cause load on the master machine, and that by proxying the WAL
> stream to the HS, using synchronous replication style to write from
> xReader to the HS, you could use the HS for a source for that data
> with it being at exactly the right point in time to query it.
Yes, that does make sense for some workloads. I don't think its viable for
everything though, thats why were not aiming for that ourselves atm.

> I'm not convinced that I would rather see the logic fixed inside the
> master as opposed to being deployable on the master's machine, the
> slave machine, or even on its own machine in between.
I don't think that you can do everything apart from the master. We currently
need shared memory for coordination between the moving parts, thats why we
have it inside the master.
It also have the advantage of being easier to setup.

> > That component is layered ontop of a totally generic xlog
> > reading/parsing library that doesn't care at all where its
> > running.
> That's cool.

> > Its also used in another cluster to read the received (filtered)
> > stream.
> I don't quite follow what you're saying there.
To interpret the xlog back into something that can be used for replication you
need to read it again. After filtering we again write valid WAL, so we can use
the same library on the sending|filtering side and on the receiving side.
But thats actually off topic for this thread ;)

> > I took a *very* short glance over the current wiki description of
> > xReader and from that it seems to me it would benefit from trying
> > to make it architecturally more similar to the rest of pg.
> We're planning on using existing protocol to talk between pieces.
> Other than breaking it out so that it can run somewhere other than
> inside the server, and allowing clients to connect to xReader to
> listen to WAL events of interest, are you referring to anything
> else?
It sounds like the xReader is designed to be one multiplexing process. While
this definitely has some advantages resource-usage-wise it doesn't seem to be
fitting the rest of the design that well. The advantages might outweigh
everything else, but I am not sure about that.
Something like registering/deregistering also doesn't fit that well with the
way walsender works as far as I understand it.

Greetings,

Andres
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services