Re: Spread checkpoint sync

On Sun, Nov 21, 2010 at 04:54:00PM -0500, Greg Smith wrote:
> Ultimately what I want to do here is some sort of smarter write-behind
> sync operation, perhaps with a LRU on relations with pending fsync
> requests. The idea would be to sync relations that haven't been touched
> in a while in advance of the checkpoint even. I think that's similar to
> the general idea Robert is suggesting here, to get some sync calls
> flowing before all of the checkpoint writes have happened. I think that
> the final sync calls will need to get spread out regardless, and since
> doing that requires a fairly small amount of code too that's why we
> started with that.

For a similar problem we had (kernel buffering too much) we had success
using the fadvise and madvise WONTNEED syscalls to force the data to
exit the cache much sooner than it would otherwise. This was on Linux
and it had the side-effect that the data was deleted from the kernel
cache, which we wanted, but probably isn't appropriate here.

There is also sync_file_range, but that's linux specific, although
close to what you want I think. It would allow you to work with blocks
smaller than 1GB.

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> Patriotism is when love of your own people comes first; nationalism,
> when hate for people other than your own comes first.
> - Charles de Gaulle