| From: | Tatsuo Ishii <ishii(at)postgresql(dot)org> |
|---|---|
| To: | emond(dot)papegaaij(at)gmail(dot)com |
| Cc: | pgpool-hackers(at)lists(dot)postgresql(dot)org |
| Subject: | Re: Segfault when a watchdog sync changes the primary under running children |
| Date: | 2026-11-11 23:20:38 |
| Message-ID: | 20261112.082038.988721655042027590.ishii@postgresql.org |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgpool-hackers |
Hi Emond,
>> We hit a child segfault in pgpool-II 4.7.2 on a 3-node watchdog cluster
>> during a planned switchover of the primary. The backtrace:
>>
>> #0 read_packets_and_process (...) at protocol/pool_process_query.c:5148
>> #1 pool_process_query (...) at protocol/pool_process_query.c:299
>> #2 do_child (...) at protocol/child.c:464
>>
>> Line 5148 is FD_ISSET(MAIN(backend)->fd, &exceptmask). The slot for the
>> node returned by pool_virtual_main_db_node_id() was NULL.
>>
>> The crashing node was a watchdog standby. When the leader performed the
>> failover, the standby applied it in sync_backend_from_watchdog(). That
>> function updates Req_info->primary_node_id and main_node_id, and only then
>> kills and re-forks the children. Unlike failover(), it does not set
>> Req_info->switching.
>>
>> A child that was processing a query at that moment got the new primary from
>> pool_virtual_main_db_node_id(). Its session had no connection to that node,
>> because the node was down when the session started. Later failbacks only
>> set need_to_restart on the child, and the client's pooled connection never
>> ended.
>>
>> The attached patch sets Req_info->switching for the duration of the sync,
>> as failover() does, so children wait until the sync is done or they are
>> killed. While switching is set, register_node_operation_request() doesn't
>> signal the main process. So when requests were queued during the sync, the
>> patch raises the failover interrupt at the end.
>>
>> The code is the same on master. With the patch, the watchdog and failover
>> regression tests (003, 004, 011–015, 018, 028, 037, 125, 131) pass on
>> V4_7_STABLE.
>>
>> Best regards,
>> Emond
>
> Sorry for late response. I am looking into this now.
I have looked through the report and the patch. I think your analysis
is correct and the patch looks good to me except a minor point:
In the bew function finish_sync_backend_from_watchdog(), a global
shared memory pointer Req_info is read at:
requests_pending = Req_info->request_queue_tail != Req_info->request_queue_head;
I think we need to copy Req_info to local volatile pointer variable
and use it to prevent unwanted compiler optimization.
volatile POOL_REQUEST_INFO *req_info = Req_info;
:
:
requests_pending = req_info->request_queue_tail != req_info->request_queue_head;
Also,
Co-Authored-By: Claude Opus 5.5 <noreply(at)anthropic(dot)com>
We do not give an author credit to an AI. So the part should be
removed. Instead "Reported-by:" tag here if you like.
Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Nadav Shatz | 2026-11-12 09:41:28 | Re: Removing support for FE/BE protocol version 2 |
| Previous Message | Tatsuo Ishii | 2026-11-11 22:53:12 | Re: Removing support for FE/BE protocol version 2 |