Re: Linux memory zone reclaim

From: "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov>
To: "Greg Smith" <greg(at)2ndQuadrant(dot)com>, <pgsql-performance(at)postgresql(dot)org>
Subject: Re: Linux memory zone reclaim
Date: 2012-07-30 16:43:36
Message-ID: 501673680200002500049227@gw.wicourts.gov
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

Greg Smith <greg(at)2ndQuadrant(dot)com> wrote:

> You can tell if this is turned on like this:
>
> echo /proc/sys/vm/zone_reclaim_mode

As a data point, the benchmarks I did for some of the 9.2
scalability features does not appear to have this turned on:

# cat /proc/sys/vm/zone_reclaim_mode
0

Our Linux version:

Linux version 2.6.32.46-0.3-default (geeko(at)buildhost) (gcc version
4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP
2011-09-29 17:49:31 +0200

This has 32 cores (64 "threads" with HT) on 4 Xeon X7560 CPUs.

Intel(R) Xeon(R) CPU X7560 @ 2.27GHz

It has 256GB RAM on 4GB DIMMs, with each core controlling 2 DIMMs
and each core able to directly talk to every other core. So, it
is non-uniform, but with this arrangement it is more a matter that
there is an 8GB set of memory that is "fast" for each core and the
other 97% of RAM is all accessible at the same speed. There were
some other options for what to install on this system or how to
install it which wouldn't have kept things this tight.

> Install the numactl utility and you can see why it's made that
> decision:

We get this:

# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39
node 0 size: 65519 MB
node 0 free: 283 MB
node 1 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47
node 1 size: 65536 MB
node 1 free: 25 MB
node 2 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55
node 2 size: 65536 MB
node 2 free: 26 MB
node 3 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63
node 3 size: 65536 MB
node 3 free: 25 MB
node distances:
node 0 1 2 3
0: 10 11 11 11
1: 11 10 11 11
2: 11 11 10 11
3: 11 11 11 10

When considering a hardware purchase, it might be wise to pay close
attention to how "far" a core may need to go to get to the most
"distant" RAM.

-Kevin

In response to

Responses

Browse pgsql-performance by date

  From Date Subject
Next Message Scott Marlowe 2012-07-30 17:09:35 Re: Linux memory zone reclaim
Previous Message Peter Geoghegan 2012-07-29 15:39:21 Help me develop new commit_delay advice