Re: Bgwriter LRU cleaning: we've been going at this all wrong

On Tue, 26 Jun 2007, Heikki Linnakangas wrote:

> How much of the buffer cache do you think we should try to keep
> clean? And how large a percentage of the buffer cache do you think have
> usage_count=0 at any given point in time?

What I discovered is that most of the really bad checkpoint pause cases I
ran into involved most of the buffer cache being dirty while also having a
non-zero usage count, which left the background writer hard-pressed to
work usefully (the LRU writer couldn't do anything, and the all-scan was
writing wastefully). I was seeing >90% dirty+usage_count>0 in the really
ugly spots.

What I like about Tom's idea is that it will keep the LRU writer in the
best possible zone for that case (writing out madly right behind the LRU
sweeper as counts get to zero) while still being fine on the more normal
ones like you describe. In particular, it should cut down on how much
client backends write buffers in an overloaded case considerably.

> That will vary widely depending on your workload, of course, but keeping 1/4
> of the buffer cache clean seems like overkill to me.

What may need to happen here is to add Tom's approach, but perhaps
restrain it using the current auto-tuning LRU patch's method of estimating
how many clean buffers are needed in the near future. Particularly on
large buffer caches, the idea of getting so far ahead of the sweep that
you're looping all the away around and following right behind the clock
sweep point may be overkill, but I think it will help enormously on
smaller caches that are often very dirty.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD