Re: Weirdness with the stats collector process

On 07/25/2016 03:20 PM, Tom Lane wrote:

Matthew Musgrove <mmusgrove(at)emortgagelogic(dot)com><mailto:mmusgrove(at)emortgagelogic(dot)com> writes:

One of our instances has been behaving -- oddly. Most queries are blazing fast. It appears to just be some of the stat views that are slow.

It sounds like requests for stats updates are not getting through to the
collector. I wonder if your kernel is blocking those packets for some
reason (probably ill-considered iptables filters?). There is a test
for packet filtering conducted during postmaster startup, but we don't
repeat it after that.

Today was the first day that I have had any significant amount of time to investigate since probably the 12th. I had checked iptables before. When I saw this message I checked, double-checked and even re-ordered the iptables rules but it didn't make any difference.

I started going through the log files under /var/log again. I didn't see anything until I made it back to /var/log/messages for July 7th. I noticed some entries for system-config-network at 14:05.

I looked at ifconfig and the files in /etc/sysconfig/network-scripts but everything looked fine. I checked what ports the postmaster and stats collector were using ('lsof -i -P | grep <port>'). I'm not sure how I missed this before but they were using an IPv6 localhost port.

I asked what network changes were made on July 7th around 14:05 and was told that they disabled IPv6! I had them turn it back on and stats started working immediately.

I have since changed the config from listen_addresses = '*' to listen_addresses = '0.0.0.0' but we won't be restarting until our next maintenance window (in about a week) when they will disable IPv6 again.

Tom and Adrian, thank you both! :)

--
Matthew Musgrove