Re: Streaming replication and WAL archive interactions

On Wed, Apr 22, 2015 at 3:38 PM, Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:
> On 04/22/2015 03:30 AM, Michael Paquier wrote:
>>
>> This is going to change a behavior that people are used to for a
>> couple of releases. I would not mind having this patch do
>> "archive_mode = on during recovery" => archive only segments generated
>> by this node + the last partial segment on the old timeline at
>> promotion, without renaming to preserve backward compatible behavior.
>> If master and standby point to separate archive locations, this way
>> the operator can sort out later the one he would want to use. If they
>> point to the same location, archive_command scripts already do
>> internally such renaming, at least that's what I suspect an
>> experienced user would do, now it is true that not many people are
>> experienced in this area I see mistakes regarding such things on a
>> weekly basis... This .partial segment renaming is something that we
>> should let the archive_command manage with its internal logic.
>
>
> Currently, the archive command doesn't know if the segment it's archiving is
> partial or not, so you can't put any logic there to manage it. But if we
> archive it with the .partial suffix, then you can put logic in the
> restore_command to check for .partial files, if you really want to.

Well, now you can check as well if there is a file with the same name
already archived and append a suffix to the new file copied, keep the
two files, and then let restore_command sort things up as it wants
with the two segment files it finds.

> I feel that the best approach is to archive the last, partial segment, but
> with the .partial suffix. I don't see any plausible real-world setup where
> the current behavior would be better. I don't really see much need to
> archive the partial segment at all, but there's also no harm in doing it, as
> long as it's clearly marked with the .partial suffix.

Well, as long as it is clearly archived at promotion, even with a
suffix, I guess that I am fine... This will need some tweaking on
restore_command for existing applications, but as long as it is
clearly documented I am fine. Shouldn't this be a different patch
though?

> BTW, pg_receivexlog also uses a ".partial" file, while it's streaming WAL
> from the server. The .partial suffix is removed when the segment is
> complete. So there's some precedence to this. pg_receivexlog adds just
> ".partial" to the filename, it doesn't add any information of what portion
> of the file is valid like I suggested here, though. Perhaps we should follow
> pg_receivexlog's example at promotion too, for consistency.

Consistency here sounds good to me.
--
Michael