Skip site navigation (1) Skip section navigation (2)

Re: GDQ iimplementation

From: Jan Wieck <JanWieck(at)Yahoo(dot)com>
To: Marko Kreen <markokr(at)gmail(dot)com>
Cc: Hannu Krosing <hannu(at)2ndquadrant(dot)com>, Josh Berkus <josh(at)agliodbs(dot)com>, pgsql-cluster-hackers(at)postgresql(dot)org
Subject: Re: GDQ iimplementation
Date: 2010-05-11 15:55:58
Message-ID: 4BE97E0E.30002@Yahoo.com (view raw or flat)
Thread:
Lists: pgsql-cluster-hackers
On 5/11/2010 11:20 AM, Marko Kreen wrote:
> On 5/11/10, Jan Wieck <JanWieck(at)yahoo(dot)com> wrote:
>> On 5/11/2010 9:36 AM, Marko Kreen wrote:
>> > And although PgQ can operate with any N >= 2 segments, it queries
>> > on 2 at a time, same as Slony.  Rest are just there to give admins
>> > some safety room for "OH F*CK" moments.  With short rotation times,
>> > it starts to seem useful..
>> >
>>
>>  Agreed. The rotation time should actually reflect the longest running
>> transactions experienced on a frequent base from the application. And there
>> needs to be a safeguard against rotating over even longer running
>> transactions.
> 
> Nightly pg_dump.. ;)
> 
>>  The problem with a long running transaction is that it could have written
>> into log segment 1 before we switched to segment 2. We can only TRUNCATE
>> segment 1 after that transaction committed AND the log has been consumed by
>> everyone interested in it.
>>
>>  I am not familiar with how PgQ/Londiste do this. Slony specifically
>> remembers the highest XID in progress at the time of switching, waits until
>> the lowest XID in progress is higher than that (so all log that ever went
>> into that segment is now visible or aborted), then waits for all log in that
>> segment to be confirmed and finally truncates the log. All this time, it
>> needs to do the UNION query over both log segments.
> 
> The "highest XID" means actually "own transaction" here?
> And it's not committed yet?  That's seems to leave transactions
> that happen before it's own commit into dubious state?

One needs to tell transactions to switch log, commit, then look at the 
highest running XID after that. Any XID lower/equal to that one could 
possibly have written into the old segment.

> 
> Although you may be fine, if you don't try to minimize
> reading both tables.
> 
> PgQ does this:
> 
> Rotate:
> 1) If some consumer reads older table, don't rotate.
> 2) Set table_nr++, switch_step1 = txid_current(), switch_step2 = NULL
> 3) Commit
> 4) Set switch_step2 = txid_current() where switch_step2 IS NULL
> 5) Commit

Right, exactly like that :)

> Reader:
> 1) xmin1 = xmin of lower snapshot of batch
> 2) xmax2 = xmax of higher snapshot of batch
> 3) if xmax2 < switch_step1, read older table
> 4) if xmin1 > switch_step2, read newer table
> 5) otherwise read both

Sounds familiar. I still don't know exactly what role the 3rd log 
segment plays in that, but it sure cannot hurt.


Jan

-- 
Anyone who trades liberty for security deserves neither
liberty nor security. -- Benjamin Franklin

In response to

Responses

pgsql-cluster-hackers by date

Next:From: Jan WieckDate: 2010-05-11 16:04:56
Subject: Re: GDQ iimplementation
Previous:From: Marko KreenDate: 2010-05-11 15:20:42
Subject: Re: GDQ iimplementation

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group