Re: Streaming base backups

2011/1/11 Garick Hamlin <ghamlin(at)isc(dot)upenn(dot)edu>:
> On Mon, Jan 10, 2011 at 09:09:28AM -0500, Magnus Hagander wrote:
>> On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
>> <cedric(dot)villemain(dot)debian(at)gmail(dot)com> wrote:
>> > 2011/1/7 Magnus Hagander <magnus(at)hagander(dot)net>:
>> >> On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
>> >> <cedric(dot)villemain(dot)debian(at)gmail(dot)com> wrote:
>> >>> 2011/1/5 Magnus Hagander <magnus(at)hagander(dot)net>:
>> >>>> On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr> wrote:
>> >>>>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> >>>>>> * Stefan mentiond it might be useful to put some
>> >>>>>> posix_fadvise(POSIX_FADV_DONTNEED)
>> >>>>>>   in the process that streams all the files out. Seems useful, as long as that
>> >>>>>>   doesn't kick them out of the cache *completely*, for other backends as well.
>> >>>>>>   Do we know if that is the case?
>> >>>>>
>> >>>>> Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
>> >>>>> not already in SHM?
>> >>>>
>> >>>> I think that's way more complex than we want to go here.
>> >>>>
>> >>>
>> >>> DONTNEED will remove the block from OS buffer everytime.
>> >>
>> >> Then we definitely don't want to use it - because some other backend
>> >> might well want the file. Better leave it up to the standard logic in
>> >> the kernel.
>> >
>> > Looking at the patch, it is (very) easy to add the support for that in
>> > basebackup.c
>> > That supposed allowing mincore(), so mmap(), and so probably switch
>> > the fopen() to an open() (or add an open() just for mmap
>> > requirement...)
>> >
>> > Let's go ?
>>
>> Per above, I still don't think we *should* do this. We don't want to
>> kick things out of the cache underneath other backends, and since we
>> can't control that. Either way, it shouldn't happen in the beginning,
>> and if it does, should be backed with proper benchmarks.
>
> Another option that occurs to me is an option to use direct IO (or another
> means as needed) to bypass the cache.  So rather than kicking it out of
> the cache, we attempt just not to pollute the cache by bypassing it for cold
> pages and use either normal io for 'hot pages', or use a 'read()' to "heat"
> the cache afterward.

AFAIR, even Linus is rejecting the idea to use it seriously, except if
I shuffle in my memory.

>
> Garick
>
>>
>> I've committed the backend side of this, without that. Still working
>> on the client, and on cleaning up Heikki's patch for grammar/parser
>> support.
>>
>> --
>>  Magnus Hagander
>>  Me: http://www.hagander.net/
>>  Work: http://www.redpill-linpro.com/
>>
>> --
>> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-hackers
>

--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support