Re: problem with large maintenance_work_mem settings and

On Fri, 2006-03-10 at 09:31 -0500, Tom Lane wrote:
> Stefan Kaltenbrunner <stefan(at)kaltenbrunner(dot)cc> writes:
> > LOG: begin index sort: unique = f, workMem = 8024000, randomAccess = f
> > LOG: switching to external sort with 28658 tapes: CPU 4.18s/13.96u sec
> > elapsed 32.43 sec
> > LOG: finished writing run 1 to tape 0: CPU 173.56s/3425.85u sec elapsed
> > 3814.82 sec
> > LOG: performsort starting: CPU 344.17s/7013.20u sec elapsed 7715.45 sec
> > LOG: finished writing final run 2 to tape 1: CPU 347.19s/7121.78u sec
> > elapsed 7827.25 sec
> > LOG: performsort done (except 2-way final merge): CPU 348.25s/7132.99u
> > sec elapsed 7846.47 sec
>
> > after that the postmaster is now consuming 99% CPU for about 22 hours(!)
>
> I'll look into it, but I was already wondering if we shouldn't bound the
> number of tapes somehow. It's a bit hard to believe that 28000 tapes is
> a sane setting.

Well, since they are not actually tapes, why not?

I thought you had changed the memory settings so that the 28658 was a
maximum, not the actual number it used. In the above example we are only
using 2 runs (tapes)... so it should just read in run 1 and then run 2
into memory and complete the final merge.

Seems reasonable to limit tapes to say 10,000. 28000 tapes allows you to
sort 448 TB without any merging...

Best Regards, Simon Riggs