Re: Load Distributed Checkpoints, take 3

Heikki Linnakangas <heikki(at)enterprisedb(dot)com> writes:
> Tom Lane wrote:
>> I still think you've not demonstrated a need to expose this parameter.

> Greg Smith wanted to explicitly control the I/O rate, and let the
> checkpoint duration vary. I personally think that fixing the checkpoint
> duration is better because it's easier to tune.

> But if we only do that, you might end up with ridiculously long
> checkpoints when there's not many dirty pages. If we want to avoid that,
> we need some way of telling what's a safe minimum rate to write at,
> because that can vary greatly depending on your hardware.

As long as the minimum rate is at least 1/bgdelay, I don't think this is
a big problem.

> But maybe we don't care about prolonging checkpoints, and don't really
> need any GUCs at all. We could then just hardcode writes_per_nap to some
> low value, and target duration close to 1.0. You would have a checkpoint
> running practically all the time, and you would use
> checkpoint_timeout/checkpoint_segments to control how long it takes. I'm
> a bit worried about jumping to such a completely different regime,
> though. For example, pg_start_backup needs to create a new checkpoint,
> so it would need to wait on average 1.5 * checkpoint_timeout/segments,

Maybe I misread the patch, but I thought that if someone requested an
immediate checkpoint, the checkpoint-in-progress would effectively flip
to immediate mode. So that could be handled by offering an immediate vs
extended checkpoint option in pg_start_backup. I'm not sure it's a
problem though, since as previously noted you probably want
pg_start_backup to be noninvasive. Also, one could do a manual
CHECKPOINT command then immediately pg_start_backup if one wanted
as-fast-as-possible (CHECKPOINT requests immediate checkpoint, right?)

> and recovery would need to process on average 1.5 as much WAL as before.
> Though with LDC, you should get away with shorter checkpoint intervals
> than before, because the checkpoints aren't as invasive.

No, you still want a pretty long checkpoint interval, because of the
increase in WAL traffic due to more page images being dumped when the
interval is short.

> If we do that, we should remove bgwriter_all_* settings. They wouldn't
> do much because we would have checkpoint running all the time, writing
> out dirty pages.

Yeah, I'm not sure that we've thought through the interactions with the
existing bgwriter behavior.

regards, tom lane