Re: Standbys, txid_current_snapshot, wraparound

On Wed, Mar 28, 2012 at 10:54 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Wed, Mar 28, 2012 at 10:24 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>> On Wed, Mar 28, 2012 at 9:48 PM, Marko Kreen <markokr(at)gmail(dot)com> wrote:
>>> On Fri, Mar 23, 2012 at 08:52:40AM +0000, Simon Riggs wrote:
>>>> Master pg_controldata - OK txid_current_snapshot() - OK
>>>> Standby pg_controldata - OK txid_current_snapshot() - lower value
>>>
>>> On Skytools list is report about master with slaves, but the
>>> lower value appears on master too:
>>>
>>>  http://lists.pgfoundry.org/pipermail/skytools-users/2012-March/001601.html
>>>
>>> Cc'd original reporter too.
>>
>> Thanks. Am looking.
>
> I can't see how this could happen on the master at all.
>
> On the standby, it can happen if we skip restartpoints for about a
> couple of billion xids. Which would be a problem.
>
> More on this tomorrow.

I've not been able to recreate the problem up till now. But "knowing"
there is a bug helps develop a theory based upon the code.

CreateCheckpoint() increments the epoch on the master at the next
checkpoint after wraparound. (I'd be happier if there was an explicit
link between those two points; there's not but I can't yet see a
problem).

When the standby receives the checkpoint record, it stores the
information in 2 places:
i) directly into ControlFile->checkPointCopy
ii) and then into XLogCtl when a safe restartpoint occurs

It's possible that a safe restartpoint could be delayed. When that
happens, the XLogCtl copy grows stale. If the delay is long enough,
then the NextXid counter will increase and will eventually be higher
than the last XLogCtl, causing the epoch returned by
GetNextXidAndEpoch() to go backwards by 1. If it is delayed even
further it would wrap again and increase again by one, then decrease,
then increase. Given enough time and/or a very busy server using lots
of xids.

At the same time, when we do UpdateControlFile() the other copy of the
epoch is written to the controlfile, so the controlfile shows the
accurate value for the epoch.

So I can explain how we get two different answers from the standby,
and I can explain how the error is very hard to reproduce and
apparently transient. I can't explain anything involving the master.

The key is "what delays the restartpoint?". And the answer there is
probably bugs in index code which purport to see invalid pages when
they probably shouldn't be there. So a REINDEX would likely help.

I'll look at a patch to improve this. Definite bug and will be backpatched.

--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services