Re: Memory reporting on CentOS Linux

On 8/17/09 4:43 PM, "Scott Carey" <scott(at)richrelevance(dot)com> wrote:
>
>
> On 8/17/09 10:24 AM, "Jeremy Carroll" <jeremy(dot)carroll(at)networkedinsights(dot)com>
> wrote:
>
>> I believe this is exactly what is happening. I see that the TOP output lists
>> a
>> large amount ov VIRT & RES size being used, but the kernel does not report
>> this memory as being reserved and instead lists it as free memory or cached.
>
> Oh! I recall I found that fun behaviour Linux and thought it was a Postgres
> bug a while back. It has lot of other bad effects on how the kernel chooses
> to swap. I really should have recalled that one. Due to this behavior, I
> had initially blamed postgres for "pinning" memory in shared_buffers in the
> disk cache. But that symptom is one of linux thinking somehow that pages
> read into shared memory are still cached (or something similar).
>
> Basically, it thinks that there is more free memory than there is when there
> is a lot of shared memory. Run a postgres instance with over 50% memory
> assigned to shared_buffers and when memory pressure builds kswapd will go
> NUTS in CPU use, apparently confused. With high OS 'swappiness' value it
> will swap in and out too much, and with low 'swappiness' it will CPU spin,
> aware on one hand that it is low on memory, but confused by the large
> apparent amount free so it doesn't free up much and kswapd chews up all the
> CPU and the system almost hangs. It behaves as if the logic that determines
> where to get memory from for a process knows that its almost out, but the
> logic that decides what to swap out thinks that there is plenty free. The
> larger the ratio of shared memory to total memory in the system, the higher
> the CPU use by the kernel when managing the buffer cache.
>

Based on a little digging, I'd say that this patch to the kernel probably
alleviates the performance problems I've seen with swapping when shared mem
is high:

http://lwn.net/Articles/286472/

Other patches have improved the shared memory tracking, but its not clear if
tools like top have taken advantage of the new info available in /proc.