From: | Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> |
---|---|
To: | Simon Riggs <simon(at)2ndquadrant(dot)com> |
Cc: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Luke Lonergan <llonergan(at)greenplum(dot)com>, pgsql-patches(at)postgresql(dot)org, Martijn van Oosterhout <kleptog(at)svana(dot)org>, Qingqing Zhou <zhouqq(at)cs(dot)toronto(dot)edu>, pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: [HACKERS] Automatically setting work_mem |
Date: | 2006-04-22 03:07:07 |
Message-ID: | 200604220307.k3M377W05944@candle.pha.pa.us |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers pgsql-patches |
Where are we on this patch?
---------------------------------------------------------------------------
Simon Riggs wrote:
> On Tue, 2006-03-21 at 17:47 -0500, Tom Lane wrote:
>
> > I'm fairly unconvinced about Simon's underlying premise --- that we
> > can't make good use of work_mem in sorting after the run building phase
> > --- anyway.
>
> We can make good use of memory, but there does come a point in final
> merging where too much is of no further benefit. That point seems to be
> at about 256 blocks per tape; patch enclosed for testing. (256 blocks
> per tape roughly doubles performance over 32 blocks at that stage).
>
> That is never the case during run building - more is always better.
>
> > If we cut back our memory usage
> Simon inserts the words: "too far"
> > then we'll be forcing a
> > significantly more-random access pattern to the temp file(s) during
> > merging, because we won't be able to pre-read as much at a time.
>
> Yes, thats right.
>
> If we have 512MB of memory that gives us enough for 2000 tapes, yet the
> initial runs might only build a few runs. There's just no way that all
> 512MB of memory is needed to optimise the performance of reading in a
> few tapes at time of final merge.
>
> I'm suggesting we always keep 2MB per active tape, or the full
> allocation, whichever is lower. In the above example that could release
> over 500MB of memory, which more importantly can be reused by subsequent
> sorts if/when they occur.
>
>
> Enclose two patches:
> 1. mergebuffers.patch allows measurement of the effects of different
> merge buffer sizes, current default=32
>
> 2. reassign2.patch which implements the two kinds of resource
> deallocation/reassignment proposed.
>
> Best Regards, Simon Riggs
>
[ Attachment, skipping... ]
[ Attachment, skipping... ]
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
> choose an index scan if your joining column's datatypes do not
> match
--
Bruce Momjian http://candle.pha.pa.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
From | Date | Subject | |
---|---|---|---|
Next Message | Dhanaraj M | 2006-04-22 07:15:07 | TODO items.. |
Previous Message | Tom Lane | 2006-04-22 01:39:53 | Re: Further reduction of bufmgr lock contention |
From | Date | Subject | |
---|---|---|---|
Next Message | Simon Riggs | 2006-04-22 10:51:19 | Re: [HACKERS] Automatically setting work_mem |
Previous Message | Jeremy Drake | 2006-04-22 02:45:44 | Re: patch to have configure check if CC is intel C compiler |