Re: BUG #4575: All page cache in shared_buffers pinned (duplicated by OS, always)

I am 99.9% certian its not a fluke of "top" (or 'free'). Or a fluke with the drop_caches linux vm signal. Otherwise, the system would not spin at 100% System cpu getting no work done if an attempt to allocate memory above that threshold, the original symptom that led me down this path of investigation.

So on a 32GB machine, setting shared_buffers to 8GB, filling them up via index scans, waiting for a checkpoint and syncing to be sure the dirty pages are negligible, then trying to allocate (and use) more than 16GB of RAM will make the sytem unresponsive. Set shared_buffers to 4GB and repeat, and it takes 24GB of allocations to cause the problem. The system essentially behaves like the shared_buffer space is 2x its size, except that the more useless half of it cannot be paged out.

When I first saw this behavior, i did not connect the 8GB that the OS could not free to the 8GB that postgres was caching, and blamed linux, spending much effort with linux vm tunables. But now that they are connected, It would seem more likely to be on this side of things. After all, how can linux even know what pages postgres still has in cache? AFAIK, you aren't memory mapping pages, just read() ing them? Wouldn't the OS only know that there is memory allocated to shared space in a process, and not know those contents are mapped pages?

Alternatively, the same behavior could occur if only the first pages put in the buffer, that replace 'empty space,' are being pinned. I would need some sort of tool that can tell me what blocks are in the OS page cache and which are in postgres to distinguish between those given the symptoms.

If there are any ideas on how I could truly distinguish where this bug actually is, or further characterize it in ways that would be useful to that end, that would be great.

I should have time to set shared_buffers above 50% and see if it kills things later today -- this sort of test would not require any special cache flushing particular to linux to see.

Thanks,

Scott

________________________________________
From: Tom Lane [tgl(at)sss(dot)pgh(dot)pa(dot)us]
Sent: Thursday, December 11, 2008 5:57 AM
To: Scott Carey
Cc: pgsql-bugs(at)postgresql(dot)org
Subject: Re: [BUGS] BUG #4575: All page cache in shared_buffers pinned (duplicated by OS, always)

"Scott Carey" <scott(at)richrelevance(dot)com> writes:
> I have determined that nearly every cached page within shared_buffers is
> being pinned in memory, preventing the OS from dropping any such pages from
> its page cache.

Shouldn't you be complaining to kernel folk rather than here?
What do you think we could do about it?

(I'm not convinced that you're seeing anything except a
platform-specific vagary in how "top" counts things, anyway.)

regards, tom lane