Re: Requested WAL segment xxx has already been removed

From: wenhui qiu <qiuwenhuifx(at)gmail(dot)com>
To: Japin Li <japinli(at)hotmail(dot)com>
Cc: Fujii Masao <masao(dot)fujii(at)oss(dot)nttdata(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: Requested WAL segment xxx has already been removed
Date: 2025-07-15 08:41:56
Message-ID: CAGjGUALfTQz4aCfN38ZRtiPmtzbdmacAm=Pse4aLbxYX4j0Cjw@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

HI Japin
Thank you for your working on this.It is useful ,when a standby node has
hardware issue repaired ,wal log usually has been archived.The
wal_keep_size parameter is difficult to estimate accurately, as hardware
repair or replacement times are often unpredictable. If the machine can be
fixed in a few days, the archived WAL files are likely still available in
the archive directory.One small regret is that postgresql currently lacks a
speed limit for sending wal logs.

Thanks

On Tue, Jul 15, 2025 at 2:11 PM Japin Li <japinli(at)hotmail(dot)com> wrote:

> On Mon, 14 Jul 2025 at 20:33, Fujii Masao <masao(dot)fujii(at)oss(dot)nttdata(dot)com>
> wrote:
> > On 2025/07/14 17:08, Japin Li wrote:
> >> Hi all,
> >> I recently hit an error with our streaming replication setup:
> >> 2025-07-14 11:52:59.361
> >> CST,"replicator","",728458,"10.9.9.74:35724
> ",68747f1b.b1d8a,1,"START_REPLICATION",2025-07-14
> >> 11:52:59 CST,3/0,0,ERROR,58P01,"requested WAL segment
> >> 00000001000000000000000C has already been
> >> removed",,,,,,"START_REPLICATION 0/C000000 TIMELINE
> >> 1",,,"standby","walsender",,0
> >> It appears the requested WAL segment 00000001000000000000000C had
> >> already been
> >> archived, and I confirmed its presence in the archive directory.
> However, when
> >> the standby tried to request this file, the primary only searched for
> it in
> >> pg_wal and didn't check the archive directory. I had to manually copy
> the
> >> segment into pg_wal to get streaming replication working again.
> >> My question is: Can we make the primary automatically search the
> >> archive if
> >> restore_command is set?
> >> I found that Fujii Masao also requested this feature [1], but it
> >> seems there
> >> wasn't a consensus.
> >
> > Yeah, I still like this idea. It's useful, for example, when we want to
> > temporarily retain WAL files, such as during planned standby maintenance,
> > to avoid "requested WAL segment ... removed." error.
> >
> > Using a replication slot is one way to retain WAL files in pg_wal,
> > but it requires the pg_wal directory to be large enough to hold all
> > WAL generated during that time, which isn't always practical.
> >
>
> Agreed. Here is a patch that fixes this.
>
> --
> Regards,
> Japin Li
>

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Hayato Kuroda (Fujitsu) 2025-07-15 09:14:27 RE: pg_logical_slot_get_changes waits continously for a partial WAL record spanning across 2 pages
Previous Message Etsuro Fujita 2025-07-15 08:26:03 Odd behavior of statement triggers with transition tables on partitions