Re: BUG #17229: Segmentation Fault after upgrading to version 13

This is what we see;
postgres(at)dtord03pgm25p:/localpart0/db/postgres/13/campaign_service/data>gdb -c /localpart0/db/postgres/13/campaign_service/data/core.28085.sig11.1634328354s /usr/pgsql-13/bin/postgresGNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7Copyright (C) 2013 Free Software Foundation, Inc.License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law.  Type "show copying"and "show warranty" for details.This GDB was configured as "x86_64-redhat-linux-gnu".For bug reporting instructions, please see:<http://www.gnu.org/software/gdb/bugs/>...Reading symbols from /usr/pgsql-13/bin/postgres...Reading symbols from /usr/lib/debug/usr/pgsql-13/bin/postgres.debug...done.done.[New LWP 28085][Thread debugging using libthread_db enabled]Using host libthread_db library "/lib64/libthread_db.so.1".
warning: the debug information found in "/usr/lib/debug//usr/pgsql-11/lib/libpq.so.5.11.debug" does not match "/usr/pgsql-11/lib/libpq.so.5" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/pgsql-11/lib/libpq.so.5.11.debug" does not match "/usr/pgsql-11/lib/libpq.so.5" (CRC mismatch).
Core was generated by `postgres: campaign_service: generic_toolkit_utility bi_tools 10.110.149.55(4096'.Program terminated with signal 11, Segmentation fault.#0  __memcpy_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:18521852            mov     -12(%rsi), %rdxMissing separate debuginfos, use: debuginfo-install llvm5.0-libs-5.0.1-7.el7.x86_64 pg_partman13-4.4.0-1.rhel7.x86_64 postgresql11-libs-11.10-1PGDG.rhel7.x86_64 repmgr_13-5.2.1-1.rhel7.x86_64(gdb) bt#0  __memcpy_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1852#1  0x00007f42aee2488b in memcpy (__len=12, __src=0x7023c19c7, __dest=0x291967d) at /usr/include/bits/string3.h:51#2  gtrgm_alloc (isalltrue=<optimized out>, siglen=siglen(at)entry=12, sign=0x7023c19c7 <Address 0x7023c19c7 out of bounds>) at trgm_gist.c:82#3  0x00007f42aee25d80 in gtrgm_picksplit (fcinfo=<optimized out>) at trgm_gist.c:852#4  0x00000000008afcfa in FunctionCall2Coll (flinfo=flinfo(at)entry=0x2bf9400, collation=<optimized out>, arg1=arg1(at)entry=43095272, arg2=arg2(at)entry=140737194144480) at fmgr.c:1164#5  0x00000000004b2d03 in gistUserPicksplit (len=2, giststate=0x2bf75d8, itup=0x29193c8, v=0x7fffee76b2e0, attno=0, entryvec=0x29194e8, r=0x7f42aecd3668) at gistsplit.c:433#6  gistSplitByKey (r=r(at)entry=0x7f42aecd3668, page=page(at)entry=0x2aaad2a7b500 <Address 0x2aaad2a7b500 out of bounds>, itup=itup(at)entry=0x29193c8, len=len(at)entry=2, giststate=giststate(at)entry=0x2bf75d8,    v=v(at)entry=0x7fffee76b2e0, attno=attno(at)entry=0) at gistsplit.c:697#7  0x00000000004aa045 in gistSplit (r=r(at)entry=0x7f42aecd3668, page=page(at)entry=0x2aaad2a7b500 <Address 0x2aaad2a7b500 out of bounds>, itup=itup(at)entry=0x29193c8, len=2,    giststate=giststate(at)entry=0x2bf75d8) at gist.c:1443#8  0x00000000004aa0fe in gistSplit (r=r(at)entry=0x7f42aecd3668, page=page(at)entry=0x2aaad2a7b500 <Address 0x2aaad2a7b500 out of bounds>, itup=itup(at)entry=0x29190b8, len=<optimized out>,    giststate=giststate(at)entry=0x2bf75d8) at gist.c:1473#9  0x00000000004aa0d6 in gistSplit (r=r(at)entry=0x7f42aecd3668, page=page(at)entry=0x2aaad2a7b500 <Address 0x2aaad2a7b500 out of bounds>, itup=itup(at)entry=0x2918c08, len=<optimized out>,    giststate=giststate(at)entry=0x2bf75d8) at gist.c:1458#10 0x00000000004aa0fe in gistSplit (r=r(at)entry=0x7f42aecd3668, page=page(at)entry=0x2aaad2a7b500 <Address 0x2aaad2a7b500 out of bounds>, itup=<optimized out>, len=<optimized out>,    giststate=giststate(at)entry=0x2bf75d8) at gist.c:1473#11 0x00000000004aa3d9 in gistplacetopage (rel=0x7f42aecd3668, freespace=0, giststate=giststate(at)entry=0x2bf75d8, buffer=15844, itup=itup(at)entry=0x7fffee76c018, ntup=ntup(at)entry=1,    oldoffnum=oldoffnum(at)entry=0, newblkno=newblkno(at)entry=0x0, leftchildbuf=leftchildbuf(at)entry=0, splitinfo=splitinfo(at)entry=0x7fffee76bf60, markfollowright=markfollowright(at)entry=true,    heapRel=0x7f42aecaffb0, is_build=false) at gist.c:303#12 0x00000000004aaeda in gistinserttuples (state=state(at)entry=0x7fffee76c050, stack=stack(at)entry=0x2918a98, giststate=giststate(at)entry=0x2bf75d8, tuples=tuples(at)entry=0x7fffee76c018, ntup=ntup(at)entry=1,    oldoffnum=oldoffnum(at)entry=0, leftchild=leftchild(at)entry=0, rightchild=rightchild(at)entry=0, unlockbuf=unlockbuf(at)entry=false, unlockleftchild=unlockleftchild(at)entry=false) at gist.c:1271#13 0x00000000004ab5cf in gistinserttuple (oldoffnum=0, tuple=0x2916628, giststate=0x2bf75d8, stack=<optimized out>, state=0x7fffee76c050) at gist.c:1224#14 gistdoinsert (r=r(at)entry=0x7f42aecd3668, itup=0x2916628, freespace=freespace(at)entry=0, giststate=giststate(at)entry=0x2bf75d8, heapRel=heapRel(at)entry=0x7f42aecaffb0, is_build=is_build(at)entry=false)    at gist.c:880#15 0x00000000004abe57 in gistinsert (r=0x7f42aecd3668, values=<optimized out>, isnull=<optimized out>, ht_ctid=0x263db48, heapRel=0x7f42aecaffb0, checkUnique=<optimized out>, indexInfo=0x263d850)    at gist.c:180#16 0x000000000062be6a in ExecInsertIndexTuples (slot=slot(at)entry=0x263db18, estate=estate(at)entry=0x263d1c8, noDupErr=noDupErr(at)entry=false, specConflict=specConflict(at)entry=0x0,    arbiterIndexes=arbiterIndexes(at)entry=0x0) at execIndexing.c:393#17 0x000000000065561a in ExecInsert (mtstate=mtstate(at)entry=0x263d598, slot=0x263db18, planSlot=0x263db18, estate=estate(at)entry=0x263d1c8, canSetTag=<optimized out>) at nodeModifyTable.c:624#18 0x00000000006569d9 in ExecModifyTable (pstate=0x263d598) at nodeModifyTable.c:2246#19 0x000000000062caa2 in ExecProcNode (node=0x263d598) at ../../../src/include/executor/executor.h:248#20 ExecutePlan (execute_once=<optimized out>, dest=0x9fb620 <spi_printtupDR>, direction=<optimized out>, numberTuples=0, sendTuples=false, operation=CMD_INSERT, use_parallel_mode=<optimized out>,    planstate=0x263d598, estate=0x263d1c8) at execMain.c:1646#21 standard_ExecutorRun (queryDesc=0x29ae260, direction=<optimized out>, count=0, execute_once=<optimized out>) at execMain.c:364#22 0x00007f42af46439d in pgss_ExecutorRun (queryDesc=0x29ae260, direction=ForwardScanDirection, count=0, execute_once=<optimized out>) at pg_stat_statements.c:1045#23 0x0000000000664e17 in _SPI_pquery (tcount=0, fire_triggers=true, queryDesc=<optimized out>) at spi.c:2511#24 _SPI_execute_plan (plan=plan(at)entry=0x7fffee76c720, paramLI=paramLI(at)entry=0x0, snapshot=snapshot(at)entry=0x0, crosscheck_snapshot=crosscheck_snapshot(at)entry=0x0, read_only=read_only(at)entry=false,    fire_triggers=fire_triggers(at)entry=true, tcount=tcount(at)entry=0) at spi.c:2288#25 0x0000000000665079 in SPI_execute (src=src(at)entry=0x258c258 " \nDELETE FROM gp_metadata_gpso.tables;\nINSERT INTO gp_metadata_gpso.tables\nSELECT * FROM tables_tmp;\n", read_only=<optimized out>,    tcount=tcount(at)entry=0) at spi.c:514#26 0x00007f42af04138e in exec_stmt_dynexecute (stmt=0x25bdaf8, estate=0x7fffee76cbe0) at pl_exec.c:4429#27 exec_stmt (estate=estate(at)entry=0x7fffee76cbe0, stmt=0x25bdaf8) at pl_exec.c:2056#28 0x00007f42af0428e3 in exec_stmts (estate=0x7fffee76cbe0, stmts=0x25bdb48) at pl_exec.c:1943#29 0x00007f42af042f22 in exec_stmt_block (estate=estate(at)entry=0x7fffee76cbe0, block=block(at)entry=0x25bdb98) at pl_exec.c:1884#30 0x00007f42af0408ee in exec_stmt (estate=estate(at)entry=0x7fffee76cbe0, stmt=0x25bdb98) at pl_exec.c:1976#31 0x00007f42af042448 in plpgsql_exec_function (func=func(at)entry=0x29636f8, fcinfo=fcinfo(at)entry=0x25567b0, simple_eval_estate=simple_eval_estate(at)entry=0x0,    simple_eval_resowner=simple_eval_resowner(at)entry=0x0, atomic=atomic(at)entry=true) at pl_exec.c:610#32 0x00007f42af04d286 in plpgsql_call_handler (fcinfo=0x25567b0) at pl_handler.c:265#33 0x000000000062848f in ExecInterpExpr (state=0x25566d8, econtext=0x2556400, isnull=<optimized out>) at execExprInterp.c:675#34 0x0000000000658a1f in ExecEvalExprSwitchContext (isNull=0x7fffee76ceb7, econtext=0x2556400, state=0x25566d8) at ../../../src/include/executor/executor.h:316#35 ExecProject (projInfo=0x25566d0) at ../../../src/include/executor/executor.h:350#36 ExecResult (pstate=<optimized out>) at nodeResult.c:136#37 0x000000000062caa2 in ExecProcNode (node=0x25562f0) at ../../../src/include/executor/executor.h:248#38 ExecutePlan (execute_once=<optimized out>, dest=0x2976218, direction=<optimized out>, numberTuples=0, sendTuples=true, operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0x25562f0,    estate=0x25560c8) at execMain.c:1646#39 standard_ExecutorRun (queryDesc=0x269a448, direction=<optimized out>, count=0, execute_once=<optimized out>) at execMain.c:364#40 0x00007f42af46439d in pgss_ExecutorRun (queryDesc=0x269a448, direction=ForwardScanDirection, count=0, execute_once=<optimized out>) at pg_stat_statements.c:1045---Type <return> to continue, or q <return> to quit---#41 0x000000000079071b in PortalRunSelect (portal=portal(at)entry=0x2512588, forward=forward(at)entry=true, count=0, count(at)entry=9223372036854775807, dest=dest(at)entry=0x2976218) at pquery.c:912#42 0x0000000000791a07 in PortalRun (portal=<optimized out>, count=9223372036854775807, isTopLevel=<optimized out>, run_once=<optimized out>, dest=0x2976218, altdest=0x2976218, qc=0x7fffee76d240)    at pquery.c:756#43 0x000000000078d6c7 in exec_simple_query (query_string=<optimized out>) at postgres.c:1239#44 0x000000000078ea37 in PostgresMain (argc=<optimized out>, argv=<optimized out>, dbname=<optimized out>, username=<optimized out>) at postgres.c:4315#45 0x00000000004879d4 in BackendRun (port=<optimized out>, port=<optimized out>) at postmaster.c:4536#46 BackendStartup (port=0x248ff50) at postmaster.c:4220#47 ServerLoop () at postmaster.c:1739#48 0x0000000000718598 in PostmasterMain (argc=argc(at)entry=3, argv=argv(at)entry=0x2462420) at postmaster.c:1412#49 0x000000000048890d in main (argc=3, argv=0x2462420) at main.c:210

Thanks,Efrain J. Berdecia

On Wednesday, October 13, 2021, 03:55:25 PM EDT, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

PG Bug reporting form <noreply(at)postgresql(dot)org> writes:
> Ever since we upgraded to PostgreSQL version 13 we are sporadically getting
> segmentation faults for the following process and all existing connections
> get killed and our streaming replication process goes down.

Hmm, can you get a stack trace from the crash?  See

https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend

            regards, tom lane