BUG #14295: Hot standby crash during tsvector rebuild

From: spencer(at)whiteskycommunications(dot)com
To: pgsql-bugs(at)postgresql(dot)org
Subject: BUG #14295: Hot standby crash during tsvector rebuild
Date: 2016-08-26 07:26:58
Message-ID: 20160826072658.15676.7628@wrigleys.postgresql.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

The following bug has been logged on the website:

Bug reference: 14295
Logged by: Spencer Thomason
Email address: spencer(at)whiteskycommunications(dot)com
PostgreSQL version: 9.4.8
Operating system: Solaris 11.3 SPARC
Description:

Hello,
We have a fax server application that performs OCR and uses full text
search. We have a nightly cron job that purges the stored text for old
records to save space. The table contains approx 60K rows and the job
deletes the text from around 100-150 records each night and then updates a
tsvector column. This database is replicated to 4 hot standby slaves, 2 in
the same data center and 2 in a remote location. We have observed a few
crashes of all the hot standby server when this cron job runs. All replicas
crash at the same time with the following log entries below. The master
appears unaffected and continues to function normally. Unfortunately I'm
unable to reproduce at will but I have observed this with 9.5.3 then we
downgraded to 9.4.8 and the crash just happened again.

Logs from one of the replicas is below:
2016-08-26 06:01:50 UTC FATAL: unexpected GIN leaf action: 0
2016-08-26 06:01:50 UTC CONTEXT: xlog redo Insert item, node:
1663/16387/33108 blkno: 6622 isdata: T isleaf: T 3 segments: 2 (add 0 items)
0 unknown action 0 ???
2016-08-26 06:01:50 UTC LOG: startup process (PID 19593) exited with exit
code 1
2016-08-26 06:01:50 UTC LOG: terminating any other active server
processes
2016-08-26 06:01:50 UTC WARNING: terminating connection because of crash of
another server process

The table schema is:
Column | Type | Modifiers
--------------+--------------------------+-----------
id | uuid | not null
fax_did_id | character varying(64) | not null
file_path | character varying(255) |
time_stamp | timestamp with time zone | not null
remote_id | character varying(60) |
caller_id | character varying(40) |
num_pages | integer |
ocr_text | text |
search_index | tsvector |
account_id | integer |
Indexes:
"fax_rxfax_pkey" PRIMARY KEY, btree (id)
"fax_rxfax_account_id" btree (account_id)
"fax_rxfax_fax_did_id" btree (fax_did_id)
"fax_rxfax_fax_did_id_like" btree (fax_did_id varchar_pattern_ops)
"fax_rxfax_search_index" gin (search_index)
Foreign-key constraints:
"fax_rxfax_account_id_fkey" FOREIGN KEY (account_id) REFERENCES
accounts(id) DEFERRABLE INITIALLY DEFERRED
"fax_rxfax_fax_did_id_fkey" FOREIGN KEY (fax_did_id) REFERENCES
dids(number) DEFERRABLE INITIALLY DEFERRED

Please let me know if there is any other info I can provide.

Thanks,
Spencer

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Tom Lane 2016-08-26 11:05:31 Re: BUG #14295: Hot standby crash during tsvector rebuild
Previous Message Pablo Pumarino Delgado 2016-08-25 15:58:42 Re: BUG #14294: Problem in generate series between dates