Load Distributed Checkpoints test results

Here's results from a batch of test runs with LDC. This patch only
spreads out the writes, fsyncs work as before. This patch also includes
the optimization that we don't write buffers that were dirtied after
starting the checkpoint.

http://community.enterprisedb.com/ldc/

See tests 276-280. 280 is the baseline with no patch attached, the
others are with load distributed checkpoints with different values for
checkpoint_write_percent. But after running the tests I noticed that the
spreading was actually controlled by checkpoint_write_rate, which sets
the minimum rate for the writes, so all those tests with the patch
applied are effectively the same; the writes were spread over a period
of 1 minute. I'll fix that setting and run more tests.

The response time graphs show that the patch reduces the max (new-order)
response times during checkpoints from ~40-60 s to ~15-20 s. The change
in minute by minute average is even more significant.

The change in overall average response times is also very significant.
1.5s without patch, and ~0.3-0.4s with the patch for new-order
transactions. That also means that we pass the TPC-C requirement that
90th percentile of response times must be < average.

All that said, there's still significant checkpoint spikes present, even
though they're much less severe than without the patch. I'm willing to
settle with this for 8.3. Does anyone want to push for more testing and
thinking of spreading the fsyncs as well, and/or adding a delay between
writes and fsyncs?

Attached is the patch used in the tests. It still needs some love..

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com