| From: | Emond Papegaaij <emond(dot)papegaaij(at)gmail(dot)com> |
|---|---|
| To: | Tatsuo Ishii <ishii(at)postgresql(dot)org> |
| Cc: | pgpool-hackers(at)lists(dot)postgresql(dot)org |
| Subject: | Re: Segfault when a watchdog sync changes the primary under running children |
| Date: | 2026-11-12 21:32:20 |
| Message-ID: | CAGXsc+b9OurHztOVyJ+P06MkL-ervXpUWDw7PrtWfZm1G7v0Ww@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgpool-hackers |
Hi Tatsuo,
Thank your for taking a look at this. Here's an updated patch.
Best regards,
Emond
Op do 12 nov 2026 om 00:20 schreef Tatsuo Ishii <ishii(at)postgresql(dot)org>:
>
> Hi Emond,
>
> >> We hit a child segfault in pgpool-II 4.7.2 on a 3-node watchdog cluster
> >> during a planned switchover of the primary. The backtrace:
> >>
> >> #0 read_packets_and_process (...) at protocol/pool_process_query.c:5148
> >> #1 pool_process_query (...) at protocol/pool_process_query.c:299
> >> #2 do_child (...) at protocol/child.c:464
> >>
> >> Line 5148 is FD_ISSET(MAIN(backend)->fd, &exceptmask). The slot for the
> >> node returned by pool_virtual_main_db_node_id() was NULL.
> >>
> >> The crashing node was a watchdog standby. When the leader performed the
> >> failover, the standby applied it in sync_backend_from_watchdog(). That
> >> function updates Req_info->primary_node_id and main_node_id, and only then
> >> kills and re-forks the children. Unlike failover(), it does not set
> >> Req_info->switching.
> >>
> >> A child that was processing a query at that moment got the new primary from
> >> pool_virtual_main_db_node_id(). Its session had no connection to that node,
> >> because the node was down when the session started. Later failbacks only
> >> set need_to_restart on the child, and the client's pooled connection never
> >> ended.
> >>
> >> The attached patch sets Req_info->switching for the duration of the sync,
> >> as failover() does, so children wait until the sync is done or they are
> >> killed. While switching is set, register_node_operation_request() doesn't
> >> signal the main process. So when requests were queued during the sync, the
> >> patch raises the failover interrupt at the end.
> >>
> >> The code is the same on master. With the patch, the watchdog and failover
> >> regression tests (003, 004, 011–015, 018, 028, 037, 125, 131) pass on
> >> V4_7_STABLE.
> >>
> >> Best regards,
> >> Emond
> >
> > Sorry for late response. I am looking into this now.
>
> I have looked through the report and the patch. I think your analysis
> is correct and the patch looks good to me except a minor point:
>
> In the bew function finish_sync_backend_from_watchdog(), a global
> shared memory pointer Req_info is read at:
>
> requests_pending = Req_info->request_queue_tail != Req_info->request_queue_head;
>
> I think we need to copy Req_info to local volatile pointer variable
> and use it to prevent unwanted compiler optimization.
>
> volatile POOL_REQUEST_INFO *req_info = Req_info;
> :
> :
> requests_pending = req_info->request_queue_tail != req_info->request_queue_head;
>
> Also,
>
> Co-Authored-By: Claude Opus 5.5 <noreply(at)anthropic(dot)com>
>
> We do not give an author credit to an AI. So the part should be
> removed. Instead "Reported-by:" tag here if you like.
>
> Regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
| Attachment | Content-Type | Size |
|---|---|---|
| v2-0001-Fix-segfault-when-a-watchdog-sync-changes-the-primar.patch | text/x-patch | 4.5 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Tatsuo Ishii | 2026-11-13 01:17:20 | Re: Segfault when a watchdog sync changes the primary under running children |
| Previous Message | Nadav Shatz | 2026-11-12 09:41:28 | Re: Removing support for FE/BE protocol version 2 |