Re: Segfault when a watchdog sync changes the primary under running children

From: Emond Papegaaij <emond(dot)papegaaij(at)gmail(dot)com>
To: Tatsuo Ishii <ishii(at)postgresql(dot)org>
Cc: pgpool-hackers(at)lists(dot)postgresql(dot)org
Subject: Re: Segfault when a watchdog sync changes the primary under running children
Date: 2026-11-12 21:32:20
Message-ID: CAGXsc+b9OurHztOVyJ+P06MkL-ervXpUWDw7PrtWfZm1G7v0Ww@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgpool-hackers

Hi Tatsuo,

Thank your for taking a look at this. Here's an updated patch.

Best regards,
Emond

Op do 12 nov 2026 om 00:20 schreef Tatsuo Ishii <ishii(at)postgresql(dot)org>:
>
> Hi Emond,
>
> >> We hit a child segfault in pgpool-II 4.7.2 on a 3-node watchdog cluster
> >> during a planned switchover of the primary. The backtrace:
> >>
> >> #0 read_packets_and_process (...) at protocol/pool_process_query.c:5148
> >> #1 pool_process_query (...) at protocol/pool_process_query.c:299
> >> #2 do_child (...) at protocol/child.c:464
> >>
> >> Line 5148 is FD_ISSET(MAIN(backend)->fd, &exceptmask). The slot for the
> >> node returned by pool_virtual_main_db_node_id() was NULL.
> >>
> >> The crashing node was a watchdog standby. When the leader performed the
> >> failover, the standby applied it in sync_backend_from_watchdog(). That
> >> function updates Req_info->primary_node_id and main_node_id, and only then
> >> kills and re-forks the children. Unlike failover(), it does not set
> >> Req_info->switching.
> >>
> >> A child that was processing a query at that moment got the new primary from
> >> pool_virtual_main_db_node_id(). Its session had no connection to that node,
> >> because the node was down when the session started. Later failbacks only
> >> set need_to_restart on the child, and the client's pooled connection never
> >> ended.
> >>
> >> The attached patch sets Req_info->switching for the duration of the sync,
> >> as failover() does, so children wait until the sync is done or they are
> >> killed. While switching is set, register_node_operation_request() doesn't
> >> signal the main process. So when requests were queued during the sync, the
> >> patch raises the failover interrupt at the end.
> >>
> >> The code is the same on master. With the patch, the watchdog and failover
> >> regression tests (003, 004, 011–015, 018, 028, 037, 125, 131) pass on
> >> V4_7_STABLE.
> >>
> >> Best regards,
> >> Emond
> >
> > Sorry for late response. I am looking into this now.
>
> I have looked through the report and the patch. I think your analysis
> is correct and the patch looks good to me except a minor point:
>
> In the bew function finish_sync_backend_from_watchdog(), a global
> shared memory pointer Req_info is read at:
>
> requests_pending = Req_info->request_queue_tail != Req_info->request_queue_head;
>
> I think we need to copy Req_info to local volatile pointer variable
> and use it to prevent unwanted compiler optimization.
>
> volatile POOL_REQUEST_INFO *req_info = Req_info;
> :
> :
> requests_pending = req_info->request_queue_tail != req_info->request_queue_head;
>
> Also,
>
> Co-Authored-By: Claude Opus 5.5 <noreply(at)anthropic(dot)com>
>
> We do not give an author credit to an AI. So the part should be
> removed. Instead "Reported-by:" tag here if you like.
>
> Regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp

Attachment Content-Type Size
v2-0001-Fix-segfault-when-a-watchdog-sync-changes-the-primar.patch text/x-patch 4.5 KB

In response to

Responses

Browse pgpool-hackers by date

  From Date Subject
Next Message Tatsuo Ishii 2026-11-13 01:17:20 Re: Segfault when a watchdog sync changes the primary under running children
Previous Message Nadav Shatz 2026-11-12 09:41:28 Re: Removing support for FE/BE protocol version 2