CPU load spikes when CentOS tries to reclaim 'cached' memory

For the past few days, we've been seeing unexpected extremely high CPU spikes
in our system. We observed the following: the 'free' memory would go down to
lower than 300 MB; at that point, 'cached' slowly starts to go down, and
then CPU starts to go way up.

It's almost as if the OS was not releasing 'cached' memory fast enough for
Postgres. Is that analysis correct? Is there a way to fix this?

Here's the session:

04:58:37 load average: 2.37, free: 532, cached: 22852
04:58:57 load average: 1.91, free: 451, cached: 22859
04:59:17 load average: 1.82, free: 469, cached: 22866
04:59:57 load average: 1.57, free: 387, cached: 22884
05:00:17 load average: 3.03, free: 574, cached: 22632
05:00:37 load average: 5.51, free: 268, cached: 22267
05:00:58 load average: 21.44, free: 237, cached: 21704
05:01:18 load average: 42.98, free: 243, cached: 21061
05:01:38 load average: 63.38, free: 266, cached: 20410
05:01:58 load average: 78.69, free: 315, cached: 20135
05:02:19 load average: 89.82, free: 214, cached: 20034
05:02:39 load average: 99.06, free: 253, cached: 19873
05:02:59 load average: 105.60, free: 390, cached: 19497
05:03:20 load average: 110.10, free: 394; cached: 19290

Here are the pertinent machine and OS and Postgres details:
RAM: 32 gigs
CPU: 24 cores; Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
RAID 10
vm.swappiness=0
PostgreSQL 9.1.11 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.4.7
20120313 (Red Hat 4.4.7-3), 64-bit
Linux ps2db 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux
postgres=# SELECT name, current_setting(name), source
postgres-# FROM pg_settings
postgres-# WHERE source NOT IN ('default', 'override');
name | current_setting |
source
------------------------------+-------------------------------+----------------------
application_name | psql | client
archive_command | /bin/true |
configuration file
archive_mode | on |
configuration file
autovacuum_analyze_threshold | 50 |
configuration file
autovacuum_freeze_max_age | 800000000 |
configuration file
autovacuum_naptime | 5min |
configuration file
autovacuum_vacuum_threshold | 50 |
configuration file
bytea_output | escape |
configuration file
checkpoint_completion_target | 0.7 |
configuration file
checkpoint_segments | 128 |
configuration file
checkpoint_timeout | 15min |
configuration file
checkpoint_warning | 30s |
configuration file
client_encoding | UTF8 | client
constraint_exclusion | partition |
configuration file
cpu_index_tuple_cost | 0.005 |
configuration file
cpu_operator_cost | 0.0025 |
configuration file
cpu_tuple_cost | 0.01 |
configuration file
custom_variable_classes | pg_stat_statements |
configuration file
DateStyle | ISO, MDY |
configuration file
default_statistics_target | 100 |
configuration file
default_text_search_config | pg_catalog.english |
configuration file
effective_cache_size | 16GB |
configuration file
effective_io_concurrency | 1 |
configuration file
enable_material | off |
configuration file
escape_string_warning | on |
configuration file
hot_standby | on |
configuration file
lc_messages | C |
configuration file
lc_monetary | en_US.UTF-8 |
configuration file
lc_numeric | en_US.UTF-8 |
configuration file
lc_time | en_US.UTF-8 |
configuration file
listen_addresses | * |
configuration file
log_autovacuum_min_duration | 0 |
configuration file
log_checkpoints | on |
configuration file
log_connections | on |
configuration file
log_destination | csvlog |
configuration file
log_directory | pg_log |
configuration file
log_disconnections | on |
configuration file
log_filename | postgresql.log.ps2db.%H |
configuration file
log_line_prefix | %t [%d] [%u] [%p]: [%l-1] %h |
configuration file
log_lock_waits | on |
configuration file
log_min_duration_statement | 0 |
configuration file
log_rotation_age | 1h |
configuration file
log_temp_files | 0 |
configuration file
log_timezone | Canada/Pacific | environment
variable
log_truncate_on_rotation | on |
configuration file
logging_collector | on |
configuration file
maintenance_work_mem | 1GB |
configuration file
max_connections | 500 |
configuration file
max_locks_per_transaction | 512 |
configuration file
max_stack_depth | 2MB | environment
variable
max_standby_streaming_delay | 90min |
configuration file
max_wal_senders | 6 |
configuration file
pg_stat_statements.max | 10000 |
configuration file
pg_stat_statements.track | all |
configuration file
port | 5432 |
configuration file
random_page_cost | 4 |
configuration file
shared_buffers | 6GB |
configuration file
shared_preload_libraries | pg_stat_statements |
configuration file
standard_conforming_strings | off |
configuration file
stats_temp_directory | /ram_postgres_stats |
configuration file
temp_buffers | 16MB |
configuration file
TimeZone | Canada/Pacific | environment
variable
wal_keep_segments | 64 |
configuration file
wal_level | hot_standby |
configuration file
work_mem | 8MB |
configuration file
(65 rows)

--
View this message in context: http://postgresql.1045698.n5.nabble.com/CPU-load-spikes-when-CentOS-tries-to-reclaim-cached-memory-tp5806122.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.