Re: pg_rewind does not rewind diverging timelines

On 7/29/26 13:51, Ants Aasma wrote:
> On Mon, 27 Jul 2026 at 02:36, Michael Paquier <michael(at)paquier(dot)xyz> wrote:
> >
> > On Mon, Jun 01, 2026 at 03:30:58PM +0900, Kyotaro Horiguchi wrote:
> > > I wonder whether strengthening the history-based matching would be
> > > sufficient instead. If timelines with the same TLI but different
> > > histories can be treated as distinct and pg_rewind continues walking
> > > the history chain until it finds a common ancestor, that seems like a
> > > fairly natural fit with the existing timeline model.
> >
> > Yeah, perhaps there is something that we could do here better in
> > pg_rewind in terms of TLI history.  A second software layer to ensure
> > TLI unicity feels just like a shortcut: we already have a LSN.
>
> The LSN can easily match for two promotions.
>
> > > UUIDs would certainly make identification straightforward, although
> > > they would also introduce longer identifiers that are a bit less
> > > convenient for humans to work with. My initial thought is that it may
> > > be worth exploring how far we can get with the existing history
> > > information before introducing a new identifier.
> >
> > This.  For this reason, I am not convinced that this proposal is
> > neither acceptable or something that we need to do at all.
> >
> > Why should we bear the burden of introducing a second level of
> > timeline identification knowing that by design we have to rely on a
> > *single* archive location for a new timeline selection when a standby
> > is triggered for promotion?  My opinion is that this is trying to fix
> > a problem for something that it not actually a problem.  If you play
> > with HA scenarios where there is a risk of two standbys reusing the
> > same timeline number, just don't do that.  We've historically claimed
> > that the archive strategy is wrong if your deployments cannot
> > guarantee a unique TLI assignment.  A new sub-identification system
> > will not provide more guarantees.
>
> The trouble is that with the current archiving model it is impossible
> to ensure a unique TLI assignment. Restore command checks for
> existence, the first identifier that gives an error will get picked,
> the promotion finished and then later it is asynchronously archived.
> If there is a failure between promotion and archiving you will get
> multiple nodes on the same timeline. There's also the issue that
> restore command has no way of distinguishing whether the archive is
> unavailable or the requested file is missing.
>
> Having two failures so close by might sound improbable, but my
> experience shows that failures are very often correlated. E.g. a gray
> failure of networked storage can cause systems to become unresponsive
> to the extent of being indistinguishable from dead, and this often
> happens in a rolling manner. Primary fails, secondary promotes and
> then immediately fails itself. I have seen this happen almost daily
> when someone misconfigures their backups to happen at the same time
> across their fleet.

These are good examples of how things can go wrong, but even if they
were uncommon, having an extra check that you're not using a bad
timeline is beneficial. Also, failures like this have a tendency to
happen in bursts when something breaks, and having everything failing at
the same time will be problematic to deal with. Imagine you have a link
failure and a bunch of servers happen to pick different timelines and
make some changes before failing over again. Trying to sort out that
kind of situation would be a nightmare.

>
> However I am not sure if this proposal is going far enough. The goal
> here is to uniquely identify transaction log to originate from a
> specific postgres instance running as a primary. It is reasonable to
> expect that HA systems ensure that for each instance running as
> primary there exists a single promote call. But not really much else.

This is true, but it is a very difficult problem to ensure that you have
consensus on who should do that promote.

> This proposal only handles the history file uniqueness in pg_rewind.
> Unless the archive command is very carefully managed and the archive
> itself is fully consistent, it is also possible to get WAL files in
> the archive from different primaries but with the same TLI. If things
> go well this might just result in replication failing, but if things
> align in the wrong way it can also cause silent data corruption. This
> is still too fragile for my taste.

I'm not sure about the scenario you are thinking about here. You can
have different primaries that write to the archive, yes, but when you
restore they should come from one or the other. They should be
"internally consistent" in the sense that you are not mixing, for
example, different versions of WALs from different primaries. If you do
that, a recovery will be impossible anyway, so that would be a
different, but quite serious, problem. If you write different versions
of timelines with same TLI that should still work since the UUID would
ensure that you are not using the wrong version of the timeline.

It might be hard to *avoid* the problem entirely, but it should not be
possible to cause a data corruption (but I might misunderstand what you
had in mind).

>
> One straightforward solution is to replace the whole TLI with a UUID
> and invent some new method of determining "latest" timeline. This has
> the obvious problem of breaking almost all of the existing tooling.
> But a less invasive way to at least detect issues and give backup
> tools ways to solve the problem would be to use this UUID proposal and
> also include it in XLogLongPageHeaderData. Recovery can then validate
> that the WAL file is from the correct timeline.

I avoided making the UUID first class to not disrupt the existing
tooling too much, but that was only the reason. One of the earlier
versions also had the UUID in the WAL, but I skipped that since it was
quite intrusive and making modifications to the format of the WAL has
its own set of problems and can break things quite seriously. At least
that was my reasoning.

Best wishes,
Mats Kindahl, Multigres team, Supabase

>
> Regards,
> Ants Aasma
>
>