Re: Segfault when a watchdog sync changes the primary under running children

From: Tatsuo Ishii <ishii(at)postgresql(dot)org>
To: emond(dot)papegaaij(at)gmail(dot)com
Cc: pgpool-hackers(at)lists(dot)postgresql(dot)org
Subject: Re: Segfault when a watchdog sync changes the primary under running children
Date: 2026-11-11 23:20:38
Message-ID: 20261112.082038.988721655042027590.ishii@postgresql.org
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgpool-hackers

Hi Emond,

>> We hit a child segfault in pgpool-II 4.7.2 on a 3-node watchdog cluster
>> during a planned switchover of the primary. The backtrace:
>>
>> #0 read_packets_and_process (...) at protocol/pool_process_query.c:5148
>> #1 pool_process_query (...) at protocol/pool_process_query.c:299
>> #2 do_child (...) at protocol/child.c:464
>>
>> Line 5148 is FD_ISSET(MAIN(backend)->fd, &exceptmask). The slot for the
>> node returned by pool_virtual_main_db_node_id() was NULL.
>>
>> The crashing node was a watchdog standby. When the leader performed the
>> failover, the standby applied it in sync_backend_from_watchdog(). That
>> function updates Req_info->primary_node_id and main_node_id, and only then
>> kills and re-forks the children. Unlike failover(), it does not set
>> Req_info->switching.
>>
>> A child that was processing a query at that moment got the new primary from
>> pool_virtual_main_db_node_id(). Its session had no connection to that node,
>> because the node was down when the session started. Later failbacks only
>> set need_to_restart on the child, and the client's pooled connection never
>> ended.
>>
>> The attached patch sets Req_info->switching for the duration of the sync,
>> as failover() does, so children wait until the sync is done or they are
>> killed. While switching is set, register_node_operation_request() doesn't
>> signal the main process. So when requests were queued during the sync, the
>> patch raises the failover interrupt at the end.
>>
>> The code is the same on master. With the patch, the watchdog and failover
>> regression tests (003, 004, 011–015, 018, 028, 037, 125, 131) pass on
>> V4_7_STABLE.
>>
>> Best regards,
>> Emond
>
> Sorry for late response. I am looking into this now.

I have looked through the report and the patch. I think your analysis
is correct and the patch looks good to me except a minor point:

In the bew function finish_sync_backend_from_watchdog(), a global
shared memory pointer Req_info is read at:

requests_pending = Req_info->request_queue_tail != Req_info->request_queue_head;

I think we need to copy Req_info to local volatile pointer variable
and use it to prevent unwanted compiler optimization.

volatile POOL_REQUEST_INFO *req_info = Req_info;
:
:
requests_pending = req_info->request_queue_tail != req_info->request_queue_head;

Also,

Co-Authored-By: Claude Opus 5.5 <noreply(at)anthropic(dot)com>

We do not give an author credit to an AI. So the part should be
removed. Instead "Reported-by:" tag here if you like.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

In response to

Responses

Browse pgpool-hackers by date

  From Date Subject
Next Message Nadav Shatz 2026-11-12 09:41:28 Re: Removing support for FE/BE protocol version 2
Previous Message Tatsuo Ishii 2026-11-11 22:53:12 Re: Removing support for FE/BE protocol version 2