Re: Scaling shared buffer eviction

On 9/11/14, 7:01 AM, Andres Freund wrote:
> I'm not convinced that these changes can be made without also changing
> the bgwriter logic. Have you measured whether there are differences in
> how effective the bgwriter is? Not that it's very effective right now :)

The current background writer tuning went out of its way to do nothing
when it wasn't clear there was something that always worked. What
happened with all of the really clever schemes was that they worked on
some workloads, and just trashed others. Most of the gain from the 8.3
rewrite came from looking at well theorized ideas for how to handle
things like pre-emptive LRU scanning for writes, and just throwing them
out altogether in favor of ignoring the problem. The magic numbers left
in or added to the code were tuned to do very little work,
intentionally. If anything, since then the pressure to do nothing has
gone up in the default install, because now people are very concerned
about extra wakeups using power.

To find bad cases before, I was running about 30 different test
combinations by the end, Heikki was running another set in the EDB lab,
I believe there was a lab at NTT running their own set too. What went in
was the combination that didn't break any of them badly--not the one
that performed best on the good workloads.

This looks like it's squashed one of the very fundamental buffer scaling
issues though; well done Amit. What I'd like to see is preserving the
heart of that while touching as little as possible. When in doubt, do
nothing; let the backends suck it up and do the work themselves.

I had to take a health break from community development for a while, and
I'm hoping to jump back into review again for the rest of the current
development cycle. I'll go back to my notes and try to recreate the
pathological cases that plagued both the 8.3 BGW rewrite and the aborted
9.2 fsync spreading effort I did; get those running again and see how
they do on this new approach. I have a decent sized 24 core server that
should be good enough for this job. I'll see what I can do.

--
Greg Smith greg(dot)smith(at)crunchydatasolutions(dot)com
Chief PostgreSQL Evangelist - http://crunchydatasolutions.com/