Re: checkpointer continuous flushing

From: Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr>
To: Andres Freund <andres(at)anarazel(dot)de>
Cc: Heikki Linnakangas <hlinnaka(at)iki(dot)fi>, PostgreSQL Developers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: checkpointer continuous flushing
Date: 2015-08-11 15:15:22
Message-ID: alpine.DEB.2.10.1508110847150.1749@sto
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Hello Andres,

> [...] Right. At the cost of keeping track of all files...

Sure. Pg already tracks all files, and probably some more tracking would
be necessary for an early fsync feature to know what are those already
fsync'ed and what are those not yet fsync'ed.

> Yes, the other version has a higher space overhead.

Yep, this is my concern.

> I'm not convinced that's meaningful in comparison to shared buffers in
> space. And rather doubtful it a loss performance wise in a loaded
> server. All the buffer headers are touched on other cores and doing the
> sort with indirection will greatly increase bus traffic.

The measures I collected and reported showed that the sorting time is
basically insignificant, so bus traffic induced by sorting does not seem
to be an issue.

> [...] It's not particularly desirable to have a performance feature that
> works less well if the server is heavily and concurrently loaded. The
> likelihood of bogus sort results will increase with the churn rate in
> shared buffers.


In conclusion I'm not convinced that it is worth the memory, but I'm also
tired of arguing, and hopefully nobody else cares about a few more bytes
per shared_buffers, so why should I care?

Here is a v8, I reduced the memory overhead of the "heavy weight" approach
from 24 to 16 bytes per buffer, so it is medium weight:-). It might be
compacted further down to 12 bytes by combining the 2 bits of forkNum
either with relNode or blockNum, and use a uint64_t comparison field with
all data so that the comparison code would be simpler and faster.
I also fixed the computation of the shmem size which I had not updated
when switching to shmem.

The patches still include the two guc, but it is easy to remove one or the
other. They are useful is someone wants to test. The default is on for
sort, and off for flush. Maybe it should be on for both.


Attachment Content-Type Size
checkpoint-continuous-flush-8-a.patch text/x-diff 20.6 KB
checkpoint-continuous-flush-8-b.patch text/x-diff 28.5 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Simon Riggs 2015-08-11 15:28:13 Re: 64-bit XIDs again
Previous Message Daniel Verite 2015-08-11 15:13:03 Re: [patch] A \pivot command for psql