Re: Re: xReader, double-effort (was: Temporary tables under hot standby)

> Aakash, when you get a chance, could you fill in the "inch-stones"
> from the GSoC proposal page onto the Wiki page?

Sure, http://wiki.postgresql.org/wiki/XReader updated.

On Sat, Apr 28, 2012 at 3:48 AM, Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov
> wrote:

> Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>
> > In the current, prototypal, state there is one component thats
> > integrated into the server (because it needs information thats
> > only available there).
>
> The xReader design was based on the idea that it would be nice not
> to cause load on the master machine, and that by proxying the WAL
> stream to the HS, using synchronous replication style to write from
> xReader to the HS, you could use the HS for a source for that data
> with it being at exactly the right point in time to query it.
>
> I'm not convinced that I would rather see the logic fixed inside the
> master as opposed to being deployable on the master's machine, the
> slave machine, or even on its own machine in between.
>
> > That component is layered ontop of a totally generic xlog
> > reading/parsing library that doesn't care at all where its
> > running.
>
> That's cool.
>
> > Its also used in another cluster to read the received (filtered)
> > stream.
>
> I don't quite follow what you're saying there.
>
> > I plan to submit the XLogReader (thats what its called atm)
> > before everything else, so everybody can take a look as soon as
> > possible.
>
> Great! That will allow more discussion and planning.
>
> > I took a *very* short glance over the current wiki description of
> > xReader and from that it seems to me it would benefit from trying
> > to make it architecturally more similar to the rest of pg.
>
> We're planning on using existing protocol to talk between pieces.
> Other than breaking it out so that it can run somewhere other than
> inside the server, and allowing clients to connect to xReader to
> listen to WAL events of interest, are you referring to anything
> else?
>
> > I also would suggest reviewing how the current walreceiver/sender,
> > and their protocol, work.
>
> Of course! The first "inch-stone" in the GSoC project plan
> basically consists of creating an executable that functions as a
> walreceiver and a walsender to just pass things through from the
> master to the slave. We build from there by allowing clients to
> connect (again, over existing protocol) and register for events of
> interest, and then recognizing different WAL records to generate
> events. The project was just going to create a simple client to
> dump the information to disk, but with the time saved by adopting
> what you've already done, that might leave more time for generating
> a useful client.
>
> Aakash, when you get a chance, could you fill in the "inch-stones"
> from the GSoC proposal page onto the Wiki page? I think the
> descriptions of those interim steps would help people understand
> your proposal better. Obviously, some of the particulars of tasks
> and the dates may need adjustment based on the new work which is
> expected to appear before you start, but what's there now would be a
> helpful reference.
>
> -Kevin
>