Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

I didn't see where the path pg_ctl

$./pg_ctl -D /data/postgres/data -l logfile -c start
-su: ./pg_ctl: No such file or directory
$ pg_ctl -D /data/postgres/data -l logfile -c start
pg_ctl: command not found

on datadir I did not see core.PID, let me know where the dir.
$ ls -lrth
total 144K
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_twophase
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_tblspc
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_snapshots
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_serial
drwx------ 4 postgres postgres 4.0K Jan 2 16:34 pg_multixact
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_dynshmem
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_commit_ts
-rw------- 1 postgres postgres 3 Jan 2 16:34 PG_VERSION
-rw------- 1 postgres postgres 88 Jan 2 16:34 postgresql.auto.conf
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_xact
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_subtrans
drwx------ 6 postgres postgres 4.0K Jan 2 16:44 base
drwx------ 5 postgres postgres 4.0K Jan 3 18:29 pg_replslot
-rw------- 1 postgres postgres 127 Jan 15 12:38 postmaster.opts
drwx------ 2 postgres postgres 4.0K Jan 16 01:00 pg_log
-rw------- 1 postgres postgres 47 Jan 16 01:00 current_logfiles
drwx------ 3 postgres postgres 52K Jan 16 11:00 pg_wal
drwx------ 4 postgres postgres 4.0K Jan 16 11:40 pg_logical
-rw------- 1 postgres postgres 93 Jan 16 11:40 postmaster.pid
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat_tmp
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_notify
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 global
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat
$ cd postmaster.pid
-su: cd: postmaster.pid: Not a directory
$ more postmaster.pid
15803
/data/postgres/data
1579063112
5432
/var/run/postgresql
*
5432001 169312256
ready

________________________________
From: Mahendra Singh Thalor <mahi6run(at)gmail(dot)com>
Sent: Wednesday, January 15, 2020 7:30 PM
To: Pendekar Dikala Senja <pendekar(dot)senja(at)outlook(dot)com>
Cc: Michael Paquier <michael(at)paquier(dot)xyz>; pgsql-bugs(at)lists(dot)postgresql(dot)org <pgsql-bugs(at)lists(dot)postgresql(dot)org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Thu, Jan 16, 2020, 08:50 Pendekar Dikala Senja <pendekar(dot)senja(at)outlook(dot)com<mailto:pendekar(dot)senja(at)outlook(dot)com>> wrote:
following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.

sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.

Yes, every time pid will change. You have to give '-c" option while stating the server to get call stack.
You have to give "gdb postgres data/core.pid".
This command should be fired from your bin(binray) folder.
Only running process can be attached like "gdb -p pid'.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com<http://www.enterprisedb.com/>

(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
Copying output to debuglog.txt.
(gdb) cont
The program is not being run.
(gdb) detach
The program is not being run.
(gdb) b errfinish
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb)

thanks

________________________________
From: Pendekar Dikala Senja <pendekar(dot)senja(at)outlook(dot)com<mailto:pendekar(dot)senja(at)outlook(dot)com>>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run(at)gmail(dot)com<mailto:mahi6run(at)gmail(dot)com>>
Cc: Michael Paquier <michael(at)paquier(dot)xyz<mailto:michael(at)paquier(dot)xyz>>; pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org> <pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org>>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,
right now we're still trying to setup again, and will provide log later.

Kind Regards

your good friend
________________________________
From: Mahendra Singh Thalor <mahi6run(at)gmail(dot)com<mailto:mahi6run(at)gmail(dot)com>>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar(dot)senja(at)outlook(dot)com<mailto:pendekar(dot)senja(at)outlook(dot)com>>
Cc: Michael Paquier <michael(at)paquier(dot)xyz<mailto:michael(at)paquier(dot)xyz>>; pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org> <pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org>>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar(dot)senja(at)outlook(dot)com<mailto:pendekar(dot)senja(at)outlook(dot)com>> wrote:
>
>
> Here are the logs and our configuration.
> Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
> As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
>
> LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.6 LTS
> Release: 16.04
> Codename: xenial
>
> Can you help us to find out the solution and configuration?
> Thanks for your help
>
> ________________________________ From: Michael Paquier
> Sent: Tuesday, January 14, 2020 6:58 PM
> To: pendekar(dot)senja(at)outlook(dot)com<mailto:pendekar(dot)senja(at)outlook(dot)com>; pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org>
> Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > I got error when replicate with native logical method from PostgreSQL 11.6
> > to PostgreSQL 12.1, I will connecting table partition to table partition
> > from Version 11.6 to Version 12.1, but after all tables already replicated
> > around more than 2 hours, server can't access and recovery mode. please help
> > to advice, below log.
>
> Could you provide a backtrace? Looking at the recent commit history,
> we had one fix which could be relevant to you after 12.1 was released:
> https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f(at)postgresql(dot)org
>
> And here is the commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
>
> The problem may be different though, so a backtrace would be good to
> check anyway.
> --
> Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--