Re: checkpointer continuous flushing

On 2015-06-02 15:42:14 +0200, Fabien COELHO wrote:
> >>This version seems already quite effective and very light. ISTM that
> >>adding a sort phase would mean reworking significantly how the
> >>checkpointer processes pages.
> >
> >Meh. The patch for that wasn't that big.
>
> Hmmm. I think it should be implemented as Tom suggested, that is per chunks
> of shared buffers, in order to avoid allocating a "large" memory.

I don't necessarily agree. But that's really just a minor implementation
detail. The actual problem is sorting & fsyncing in a way that deals
efficiently with tablespaces, i.e. doesn't write to tablespaces
one-by-one. Not impossible, but it requires some thought.

> >The problem with doing this separately is that without the sorting this
> >will be slower for throughput in a good number of cases. So we'll have
> >yet another GUC that's very hard to tune.
>
> ISTM that the two aspects are orthogonal, which would suggests two gucs
> anyway.

They're pretty closely linked from their performance impact. IMO this
feature, if done correctly, should result in better performance in 95+%
of the workloads and be enabled by default. And that'll not be possible
without actually writing mostly sequentially.

It's also not just the sequential writes making this important, it's
also that it allows to do the final fsync() of the individual segments
as soon as their last buffer has been written out. That's important
because it means the file will get fewer writes done independently
(i.e. backends writing out dirty buffers) which will make the final
fsync more expensive.

It might be that we want to different gucs, but I don't think we can
release without both features.

Greetings,

Andres Freund