Performance issue - 2 linux machines, identical configs, different performance

Hi, sorry about the blank post yesterday - let's try again

We have two machines. Both running Linux Redhat, both running postgres
8.2.5.

Both have nearly identical 125 GB databases. In fact we use PITR Recovery
to

Replicate from one to the other. The machine we replicate to runs a query
with

About 10 inner and left joins about 5 times slower than the original machine

I run an explain on both. Machine1 (original) planner favors hash joins
about 3 to 1

Over nested loop joins. Machine2 (replicated) uses only nested loop joins -
no hash at all.

A few details - I can always provide more

MACHINE1 - original:

TOTAL RAW MEMORY - 30 GB

TOTAL SHARED MEMORY (shmmax value) - 4 GB

Database configs

SHARED_BUFFERS ------------- 1525 MB

MAX_PREPARED_TRANSACTIONS - 5

WORK_MEM - -------------------- 300 MB

MAINTENANCE_WORK_MEM - 512 MB

MAX_FSM_PAGES -------------- 3,000,000

CHECKPOINT_SEGMENTS ----- 64

WAL_BUFFERS --------------------- 768

EFFECTIVE_CACHE_SIZE ---- 2 GB

Planner method configs all turned on by default, including
enable_hashjoin

MACHINE2 - we run 2 postgres instances. Port 5433 runs continuous PITR
recoveries

Port 5432 receives the 'latest and greatest' database when port 5433
finishes a recovery

TOTAL RAW MEMORY - 16 GB (this is a VMWARE setup on a netapp)

TOTAL SHARED MEMORY (shmmax value) - 4 GB

Database configs - port 5432 instance

SHARED_BUFFERS ------------ 1500 MB

MAX_PREPARED_TRANSACTIONS - 1 (we don't run prepared transactions
here)

WORK_MEM - -------------------- 300 MB

MAINTENANCE_WORK_MEM - 100 MB (don't think this comes into play
in this conversation)

MAX_FSM_PAGES -------------- 1,000,000

CHECKPOINT_SEGMENTS ----- 32

WAL_BUFFERS --------------------- 768

EFFECTIVE_CACHE_SIZE ---- 2 GB

Planner method configs all turned on by default, including
enable_hashjoin

Database configs - port 5433 instance

SHARED_BUFFERS ------------ 1500 MB

MAX_PREPARED_TRANSACTIONS - 1 (we don't run prepared transactions
here)

WORK_MEM - -------------------- 250 MB

MAINTENANCE_WORK_MEM - 100 MB (don't think this comes into play
in this conversation)

MAX_FSM_PAGES -------------- 1,000,000

CHECKPOINT_SEGMENTS ----- 32

WAL_BUFFERS --------------------- 768

EFFECTIVE_CACHE_SIZE ---- 2 GB

Planner method configs all turned on by default, including
enable_hashjoin

Now some size details about the 11 tables involved in the join

All join fields are indexed unless otherwise noted and are of type
integer unless otherwise noted

TABLE1 -------------398 pages

TABLE2 -------- 5,014 pages INNER JOIN on TABLE1

TABLE3 ------- 34,729 pages INNER JOIN on TABLE2

TABLE4 ----1,828,000 pages INNER JOIN on TABLE2

TABLE5 ----1,838,000 pages INNER JOIN on TABLE4

TABLE6 ------ 122,500 pages INNER JOIN on TABLE4

TABLE7 ----------- 621 pages INNER JOIN on TABLE6

TABLE8 ---------- 4 pages INNER JOIN on TABLE7 (TABLE7 column
not indexed)

TABLE9 ----------- 2 pages INNER JOIN on TABLE8 (TABLE8 column
not indexed)

TABLE10 --------- 13 pages LEFT JOIN on TABLE6 (columns on both
tables text, neither column indexed)

TABLE11 -1,976,430 pages LEFT JOIN on TABLE5. AND explicit join on
TABLE6

The WHERE clause filters out primary key values from TABLE1 to 1
value and a 1 month range of

Indexed dates from TABLE4.

So, my guess is the disparity of performance (40 seconds vs 180 seconds)
has to do with MACHINE2 not

Availing itself of hash joins which by my understanding is much faster.

Any help / insight appreciated. Thank you

Mark Steben│Database Administrator│

@utoRevenue-R- "Join the Revenue-tion"
95 Ashley Ave. West Springfield, MA., 01089
413-243-4800 x1512 (Phone) │ 413-732-1824 (Fax)

@utoRevenue is a registered trademark and a division of Dominion Enterprises