Re: Add timeline to partial WAL segments

On 12/12/18 7:17 PM, Michael Paquier wrote:
> On Wed, Dec 12, 2018 at 07:54:05AM -0500, David Steele wrote:

>> But, we could at least use the . notation and end up with something like
>> 000000010000000100000001.00000002.partial or perhaps
>> 000000010000000100000001.T00000002.partial? Maybe
>> 000000010000000100000001.00000002.tpartial?
>>
>> I can't decide whether the .partial files generated by pg_receivewal are
>> a problem or not. It seems to me that only the original cluster is
>> going to be able to stream this file -- once the segment is renamed to
>> .partial it won't be visible to pg_receivexlog. So, .partial without a
>> timeline extension just means partial from the original timeline.
>
> Still this does not actually solve the problem when two servers are
> trying to use the same timeline? You would get conflicts with the
> history file as well in this case but the partial segment gets archived
> first.. It seems to me that it is kind of difficult to come with a
> totally bullet-proof solution.

What we would like to do here is reduce the window in which the server
will make a bad choice by pushing timeline files first, then allow the
server to archive by using a more unique name for partial WAL.

It is understood that servers will need to be rebuilt in HA
environments, but we would like to reduce the frequency. More
importantly we want to preserve the integrity of the WAL archive since
it serves as the ground truth for all the clusters. Reducing duplicates
spares users decisions about what to delete to get their cluster
archiving again.

>> There's another difference. The .partial generated by pg_receivewal is
>> an actively-worked-on file whereas the .partial generated by a promotion
>> is probably headed for oblivion. I haven't see a single case where one
>> was used in an actual recovery (which doesn't mean it hasn't happened,
>> of course).
>
> There are many people implementing their own backup solutions, it is
> hard to say that none of those solutions are actually able to copy the
> last partial file to replay up to the end of a wanted timeline for a
> PITR.

Indeed, we've thought about adding it to pgBackRest. But we haven't,
and to my knowledge nobody else has either. It will come though, and a
better way of naming these files will help make them useful.

Regards,
--
-David
david(at)pgmasters(dot)net