Re: 028_pitr_timelines.pl might fail when cp interrupted and backup includes required segment

From: Andrey Borodin <x4mmm(at)yandex-team(dot)ru>
To: Alexander Lakhin <exclusion(at)gmail(dot)com>
Cc: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: 028_pitr_timelines.pl might fail when cp interrupted and backup includes required segment
Date: 2026-08-23 10:27:32
Message-ID: A0F42538-D49C-4A2E-B786-96085C221DEA@yandex-team.ru
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi Alexander,

On Sat, Aug 22, 2026, Alexander Lakhin wrote:
> That is, cp was interrupted mid-flight, probably due to slow IO at
> that moment.
>
> As it turned out, under normal conditions, the standby instance
> overwrites segment 0002 in the archive on $node_standby->stop:

Thanks for tracking this down. I think this points to a general issue in
Test::Cluster rather than another race specific to 028_pitr_timelines.
The archive command publishes the final file name while cp is still
writing it. The test then happens to rely on the standby overwriting a
partial file left by the immediate shutdown.

I propose making archive publication more atomic. The attached patch makes
the archive command follow this sequence:

copy source temporary
mv temporary archive-file

Test::Cluster generates a random component for the temporary name, and
the shell PID further distinguishes concurrent Unix commands. The move
publishes the completed file atomically and retains the existing overwrite
behavior. On both platforms, a failed copy remains invisible to
restore_command.

On Unix we could instead use ln followed by cmp to avoid replacing an
existing archive file with different contents. I did not do that here because
it would change the semantics of Test::Cluster's archive command. WAL-G also
overwrites WAL by default. With WALG_PREVENT_WAL_OVERWRITE enabled, it first
downloads an existing object and compares it with the local WAL, although
there is still a race between that check and the upload. As a maintainer of a
backup tool, I would prefer more PostgreSQL tests to exercise the default
overwrite semantics used by such tools. For end user it is twice cheaper in
calls to S3 (PUT vs GET+PUT).

WDYT?

Thank you!

Best regards, Andrey Borodin.

Attachment Content-Type Size
v1-0001-Make-test-WAL-archiving-publish-files-atomically.patch application/octet-stream 1.7 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Pavel Stehule 2026-08-23 10:31:30 Re: missing possibility to use alternative translated month names in to_char function
Previous Message cca5507 2026-08-23 09:43:55 Re: [BUG] Take a long time to reach consistent after pg_rewind