Re: GDQ iimplementation (was: Re: Clustering features for upcoming developer meeting -- please claim yours!)

On 5/11/10, Jan Wieck <JanWieck(at)yahoo(dot)com> wrote:
> I changed the subject line because we are diving deep into implementation
> details.
> On 5/11/2010 5:24 AM, Marko Kreen wrote:
> > On 5/11/10, Jan Wieck <JanWieck(at)yahoo(dot)com> wrote:
> > > The thing this event id alone does not provide is any point where
> inside
> > > that sequence of event id's the replica can issue commits. On a busy
> server,
> > > there may never be any such moment unless the replica applies things the
> > > Slony way instead of in monotonically increasing event id's. If your
> idea is
> > > to simply record things WAL style and shove them off to the replicas,
> you
> > > just move some of the current overhead from the master by duplicating it
> > > onto every replica.
> > >
> >
> > I'm not sure about what overhead are you talking about.
> >
> > Are you trying to get rid of current snapshot-based grouping
> > of events? Why?
> >
>
> The problem statement on the Wiki page and Itagaki's comments about
> non-table storage of the queue made it look to me as if some WAL style flat
> file approach was looked after.
>
> I am glad that we agree that we cannot get rid of the snapshot based
> grouping. That and the IMHO required table storage is the overhead I was
> talking about. We should be clear that we cannot get rid of that grouping
> and that however many log segments are used (Slony currently 2, Londiste
> default 3), the oldest running transaction on the master determines which
> log segments can get truncated. The more log segments there are in use, the
> more UNION keywords may appear in the query, selecting from the log.

Seems we are in agreement.

And although PgQ can operate with any N >= 2 segments, it queries
on 2 at a time, same as Slony. Rest are just there to give admins
some safety room for "OH F*CK" moments. With short rotation times,
it starts to seem useful..

There does not seem any advantage for querying more than 2 segments.

> > > There are more things to consider about such a generalized queue,
> > > especially if we think about adding it to core.
> > >
> > > One for example is version independence. Slony and I think Londiste too
> can
> > > replicate across PostgreSQL server versions. And experience shows us
> that no
> > > communications protocol, on disk format or the like is ever set in
> stone. So
> > > we need to think how this queue can become backwards compatible without
> > > introducing more overhead than we try to save right now.
> > >
> >
> > I'm guessing you are trying to do 2 more things:
> >
> > 1) Add queue operations to SQL syntax
> > 2) Non-table custom storage.
> >
>
> No. I don't know how you read 1) into the above and 2) was my
> misunderstanding reading the Wiki. I don't want either.

Oh sorry, I got that impression from wiki, not from you.

As there are some ideas from you on the wiki, I assumed
you are involved, so used 'you' very liberally.

--
marko