Several optimization options (config/hardware)

Hi,

we want to see if we can gain better performance with our postgresql
database. In the last year the amount of data growed from ~25G to now
~140G and we're currently developing a new feature that needs to get
data faster from the database. The system is both read and write heavy.

At first I want to give you an overview over the hardware, software and
configuration and the changes that I see we could check out. I'd be very
happy if you could review and tell if the one or the other is nonsense.

Hardware:
- CPU: 4x4 Cores Intel Xeon L5630 @ 2.13GHz
- RAM: 64GB
- RAID 1 (1+0) HP Company Smart Array G6 controllers, P410i
(I don't know the actual number of discs)
- A single partition for data and wal-files

Software
- RHEL 6, Kernel 2.6.32-220.4.1.el6.x86_64
- postgresql90-server-9.0.6-1PGDG.rhel6.x86_64

Configuration (selected from settings)
------------------------------+-----------+--------+-------------------
name | setting | unit | source
------------------------------+-----------+--------+-------------------
autovacuum | on | [NULL] | configuration file
checkpoint_completion_target | 0.5 | [NULL] | default
checkpoint_segments | 16 | | configuration file
checkpoint_timeout | 300 | s | default
commit_delay | 0 | | default
default_statistics_target | 100 | | default
effective_cache_size | 16384 | 8kB | default
fsync | on | [NULL] | default
log_min_duration_statement | 250 | ms | configuration file
log_temp_files | -1 | kB | default
maintenance_work_mem | 16384 | kB | default
max_connections | 2000 | | configuration file
random_page_cost | 4 | [NULL] | default
shared_buffers | 1310720 | 8kB | configuration file
synchronous_commit | on | [NULL] | default
wal_buffers | 256 | 8kB | configuration file
wal_sync_method | fdatasync | [NULL] | default
wal_writer_delay | 200 | ms | default
work_mem | 1024 | kB | default
------------------------------+-----------+--------+-------------------

Some stats:
$ free -m
total used free shared buffers cached
Mem: 64413 63764 649 0 37 60577
-/+ buffers/cache: 3148 61264
Swap: 8191 333 7858

iostat shows nearly all the time ~100% io utilization of the disc
serving the pg data / wal files.

I'd suggest the following changes:

(Improve query planning)
1) Increase effective_cache_size to 48GB
2) Increase work_mem to 10MB (alternatively first activate
log_temp_files to see if this is really needed
3) Reduce random_page_cost to 1

(WAL / I/O)
4) Set synchronous_commit=off
5) Increase checkpoint_segments to 32
6) Increase wal_buffers to 16M
7) Add new discs (RAID) for wal files / pg_xlog

(Misc)
8) Increase maintainance_work_mem to 1GB

In parallel I'd review statistics like long running queries, index usage
(which ones can be dropped) etc.

At first I'd like to try out 1) to 3) as they affect the query planner,
so that some indices that are not used right now might be used then.

After this change I'd review index usage and clean up those / improve
queries.

Then, finally I'd test WAL / I/O related changes.

Do you think this makes sense? Do you see other improvements, or do you
need some more information?

Thanx in advance,
cheers,
Martin