BUG #16351: PostgreSQL closing connection during requests with segmentation fault

From: PG Bug reporting form <noreply(at)postgresql(dot)org>
To: pgsql-bugs(at)lists(dot)postgresql(dot)org
Cc: johannes(dot)mols(at)googlemail(dot)com
Subject: BUG #16351: PostgreSQL closing connection during requests with segmentation fault
Date: 2020-04-08 08:55:34
Message-ID: 16351-5cac06fa29fc19e1@postgresql.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

The following bug has been logged on the website:

Bug reference: 16351
Logged by: Johannes Mols
Email address: johannes(dot)mols(at)googlemail(dot)com
PostgreSQL version: 10.12
Operating system: Ubuntu 18.04
Description:

I originally posted this on StackExchange yesterday, but someone recommended
me to post this here:
https://dba.stackexchange.com/questions/264530/postgresql-closing-connection-during-requests

I have a PostgreSQL 10.12 server running on an Ubuntu 18.04 Server. Since
today, I am suddenly getting errors when querying data from a Python script
running on the same machine, and in some edge cases also in pgAdmin 4.

In about 50% of the cases, my Python script will fail to return any data
because the DB server closed the connection. I am using psycopg2 and it is
giving me the following error:

psycopg2.errors.AdminShutdown: terminating connection due to administrator
command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Interestingly, the same queries (which are simple "SELECT * FROM table"
statements) run perfectly fine in pgAdmin. However, when I try to run the
following query in pgAdmin, I get the message Connection to the server has
been lost.

select * from mytable where mycolumn <> ' '

This query specifically worked fine a few days ago. I couldn't find any
other examples where this happens in pgAdmin, but there is surely something
wrong.

What is weird is that the database has been running for over a month without
any issues, and this only started today. I did try to restart the service to
no avail.

The logs are not making much sense to me, but there are several things
wrong.

The last few days, it constantly gave me the following message:

2020-04-06 19:43:26.959 CEST [7820] postgres(at)postgres FATAL: password
authentication failed for user "postgres"
2020-04-06 19:43:26.959 CEST [7820] postgres(at)postgres DETAIL: Password does
not match for user "postgres".
Connection matched pg_hba.conf line 102: "host all all
all md5"

I am certain that I am using the correct password, as it does work 50% of
the time of the queries. It is showing this message for the default postgres
database and not my own, but it still confuses me why this happens as I'm
using the default postgres user for both databases. I got this message a lot
during the night where I was not doing anything with it either.

For the actual error regarding the query from pgAdmin, this is what I get:

2020-04-07 11:25:18.891 CEST [5497] LOG: server process (PID 20569) was
terminated by signal 11: Segmentation fault
2020-04-07 11:25:18.891 CEST [5497] DETAIL: Failed process was running:
SELECT * FROM transactionoutputs WHERE fileheader <> ''
2020-04-07 11:25:18.891 CEST [5497] LOG: terminating any other active
server processes
2020-04-07 11:25:18.891 CEST [20567] postgres(at)bitcoin WARNING: terminating
connection because of crash of another server process
2020-04-07 11:25:18.891 CEST [20567] postgres(at)bitcoin DETAIL: The
postmaster has commanded this server process to roll back the current
transaction and exit, because another server process exited abnormally and
possibly corrupted shared memory.
2020-04-07 11:25:18.891 CEST [20567] postgres(at)bitcoin HINT: In a moment you
should be able to reconnect to the database and repeat your command.
I can't seem to find any errors in the logs for the regular SELECT queries
from my Python scripts, so I don't know why the connection is closed for
this all the time. What is so confusing about this is that it worked
perfectly fine just a yesterday, and I have not changed anything. Restarting
PostgreSQL also did not help.

I also constantly get this message when trying to query data through my
Python API using Postman:

2020-04-07 11:25:18.891 CEST [9215] postgres(at)bitcoin WARNING: terminating
connection because of crash of another server process
2020-04-07 11:25:18.891 CEST [9215] postgres(at)bitcoin DETAIL: The postmaster
has commanded this server process to roll back the current transaction and
exit, because another server process exited abnormally and possibly
corrupted shared memory.
2020-04-07 11:25:18.891 CEST [9215] postgres(at)bitcoin HINT: In a moment you
should be able to reconnect to the database and repeat your command.

You can see the full log of the last ~3 days here:
https://pastebin.com/iBC1G2Qg

I'm not an expert with databases or Linux, so any help why this might happen
and how I can figure out what exactly is happening is highly appreciated,
thank you.

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Jehan-Guillaume de Rorthais 2020-04-08 11:58:30 Re: [BUG] non archived WAL removed during production crash recovery
Previous Message PG Bug reporting form 2020-04-08 08:39:39 BUG #16350: pgcenter expects postgresql12-libs on postgresql 11