| From: | Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> |
|---|---|
| To: | "T, Adesh" <adeshdb(at)amazon(dot)com> |
| Cc: | PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org> |
| Subject: | Re: Postmaster PANIC on a standby with slot synchronization enabled |
| Date: | 2026-11-20 23:40:58 |
| Message-ID: | CAD21AoBsnaSueUEXOWm36Zgmh8uCHsBW+8d4NGfnObr-brc0HA@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi,
Thank you for the report.
On Fri, Nov 20, 2026 at 2:46 PM T, Adesh <adeshdb(at)amazon(dot)com> wrote:
>
> Your attachments are now available for secure download. Amazon has replaced the attachments in this email with download links.
> Important: These links may contain content that is confidential. Please exercise appropriate caution when sharing.
> 2 Attachments Download all Passcode required VESgWX
> v2-0001-Don-t-wait-for-LogicalDecod...the-.patch v2-0002-Add-a-test-for-the-postmast...calD.patch
> Attachments Available Until: December 20, 2026, 22:46 (UTC+00:00)
> Help us improve: Share Your Feedback Visit Here for FAQs and support resources.
> Hi,
>
> On a standby with slot synchronization enabled, the postmaster can PANIC
> with "cannot wait without a PGPROC structure" and exit, taking the whole
> server down. While no slot sync worker is running, the postmaster checks
> on every wakeup whether logical decoding is enabled. Since 67c20979ce7,
> which turns logical decoding on when the first logical slot is created,
> that check takes an LWLock. If the startup process holds that lock
> exclusively while it replays a logical decoding status change, the
> postmaster has to wait, and it can't wait without a PGPROC. The window is
> narrow, but it is easy to hit with a debugger or an injection point, and
> it reproduces on current master.
Good find. It should be fixed.
> The attached 0001 has the postmaster read the flag without the lock. It is
> a single bool, and a locked read would not stay valid until the worker
> starts anyway. To cover a stale read, the slot sync worker checks again
> under the lock when it starts and exits if logical decoding is off, which
> costs at most one extra worker start. Other options would be to make the
> flag an atomic in shared memory, or to drop the check from the postmaster
> and let the worker decide. 0002 adds an injection point and a TAP test
> that reproduces the PANIC; it is not for commit. The fix needs to be
> backpatched to v19.
Regarding the proposed change, while I agree the basic idea of the
fix, I have a comment in the following change:
+ if (MyProc == NULL)
+ return LogicalDecodingCtl->logical_decoding_enabled;
I'm not sure we need to make the flag an atomic in shmem as it's a boolean flag.
I think it's better to introduce a function, say
IsLogicalDecodingEnabledNoLock() that reads the flag without lwlock so
that the caller explicitly specifies whether to read the flag without
lock. If the postmaster reads a stale value of the flag, we would end
up delaying launching the slotsync worker or launching it even if the
logical decoding is disabled. I think the former is not a problem in
practice since the startup process wakes up the postmaster after
enabling logical decoding, and the postmaster checks it again within
60 sec in the worst case. The latter case could happen even today and
even reading the flag with LogicalDecodingControlLock doesn't help
since the logical decoding can be disabled after the postmaster
launches the slotsync worker. So I think the proposed approach would
work.
Another possible fix would be to let the postmaster start the slotsync
worker without logical decoding status check, and have the slotsync
worker check it with a lock and exit. But it would require more
changes and change the existing behavior, which is not appropriate for
v19.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Robert Haas | 2026-11-20 23:50:59 | Re: Expanding HOT updates for expression and partial indexes |
| Previous Message | Melanie Plageman | 2026-11-20 23:30:04 | Re: Stop masking visibility information during WAL consistency checking |