Skip site navigation (1) Skip section navigation (2)

Re: Spec discussion: Generalized Data Queue / Modification Trigger

From: Josh Berkus <josh(at)agliodbs(dot)com>
To: Hannu Krosing <hannu(at)2ndquadrant(dot)com>
Cc: Greg Sabino Mullane <greg(at)endpoint(dot)com>, PostgresCluster ML <pgsql-cluster-hackers(at)postgresql(dot)org>
Subject: Re: Spec discussion: Generalized Data Queue / Modification Trigger
Date: 2010-03-03 21:43:43
Message-ID: 4B8ED80F.2020701@agliodbs.com (view raw or flat)
Thread:
Lists: pgsql-cluster-hackers
> I't may seem easy to replace a database table with "something else" for
> collecting the changes which have happened during the transaction, but
> you have to answer the following questions:
> 
> 1) do I need persistence, what about 2PC ?
> 
> 2) does the "something else" work well for all situations an event table
> would work (say, for example, a load of 500GB of data in one
> transaction)

Those are good questions, and a generic system would need to work for
all three of those requirements.

> 3) what would I gain in return for all the work needed to implement the
> "something else" ?

Speed.  In my test case, which was replicating view snapshots between
PostgreSQL and Redis, the difference between using an event table and
perverting the constrainttriggers to do an after-insert trigger directly
to redis was a speed difference of around 400%, not counting vacuum
overhead.

>>>> (3) A method of marking DDL changes in the data modification stream.
> 
> Yes, DDL triggers or somesuch would be highly desirable.
> 
>>> Hmm..can you expand on what you have in mind here? Something more than 
>>> just treating the DDL as another item in the (txn ordered) queue?
>> Yeah, that would be one way to handle it.  Alternately, you could have
>> the ability to mark rows with a DDL "version".
> 
> But the actual DDL would still need to be transferred, no ? 

Yes.  It may be that having a ddl change simply inserted into the
replication stream is the way to go.  Alternatively, DDL versioning
makes a certain amount of sense except that it's pretty hard to make
generic, and would require additional catalog tables.

--Josh Berkus


In response to

Responses

pgsql-cluster-hackers by date

Next:From: Hannu KrosingDate: 2010-03-03 22:12:13
Subject: Re: Spec discussion: Generalized Data Queue / Modification Trigger
Previous:From: Hannu KrosingDate: 2010-03-03 21:01:44
Subject: Re: Spec discussion: Generalized Data Queue / Modification Trigger

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group