FATAL: terminating walreceiver process due to administrator command

Hello,

Today, I received email notifications from a server telling me the
replication was lagging.

The application is monitoring the delay. In the past, it happened a few
times I received this notification during a high load but not from this
server.

However, the replication was not lagging. It stopped.

There was a single line in the log of the standby server:

FATAL: terminating walreceiver process due to administrator command

I did not find anything related in the master’s log. The rest of the log
was all about the slow statements.

As far I can tell you, this never happened before. The standby server was
running but the replication stopped. I restarted the server after 42
minutes of that line in the log. The replication caught up in 3-5 seconds.

Only I have access to the servers.

I did not stop the replication process. I don’t even know how to do it.

There is no cron task that would do such thing. Only one application access
the database. I wrote it hence I know it didn’t do it either.

I have been running the servers for years with more or less the same
configuration.

As far as I see, when I see the same line in earlier logs, the database was
shut down as well. This was the only lonely line like that.

Recent changes on the servers:

* On 11 January, I upgraded from 10.5 to 10.6_2

* A few days ago, set up a new server that replicates one table from the
same master. This is a huge table but it’s rarely written. The replication
works. It’s the same time I used logical replication. The server where the
replication stopped uses async stream replication.

* When I set up the logical replication, I increased the wal_sender_timeout

I found nothing related in the logs (/var/log/messages, /var/log/all.log,
dmesg). This slave is probably the least loaded server of the group.

FreeBSD xxx 11.2-RELEASE-p8 FreeBSD 11.2-RELEASE-p8 #0: Tue Jan 8 21:35:12
UTC 2019 root(at)amd64-builder(dot)daemonology(dot)net:/usr/obj/usr/src/sys/GENERIC
amd64

/boot/loader.conf:
# PostgreSQL
kern.ipc.semmni=256
kern.ipc.semmns=512
kern.ipc.semmnu=256

Everything else is either FreeBSD default or unrelated.

There is a lot of free memory. I don’t mean usable but free. 3GB RAM was
not even touched since the last boot.

M.