Re: [HACKERS] sort on huge table

>
>Tatsuo Ishii <t-ishii(at)sra(dot)co(dot)jp> writes:
>> I have compared current with 6.5 using 1000000 tuple-table (243MB) (I
>> wanted to try 2GB+ table but 6.5 does not work in this case). The
>> result was strange in that current is *faster* than 6.5!
>
>> RAID5
>> current 2:29
>> 6.5.2 3:15
>
>> non-RAID
>> current 1:50
>> 6.5.2 2:13
>
>> Seems my previous testing was done in wrong way or the behavior of
>> sorting might be different if the table size is changed?
>
>Well, I feel better now, anyway ;-). I thought that my first cut
>ought to have been about the same speed as 6.5, and after I added
>the code to slurp up multiple tuples in sequence, it should've been
>faster than 6.5. The above numbers seem to be in line with that
>theory. Next question: is there some additional effect that comes
>into play once the table size gets really huge? I am thinking maybe
>there's some glitch affecting performance once the temp file size
>goes past one segment (1Gb). Tatsuo, can you try sorts of say
>0.9 and 1.1 Gb to see if something bad happens at 1Gb? I could
>try rebuilding here with a small RELSEG_SIZE, but right at the
>moment I'm not certain I'd see the same behavior you do...

Ok. I have run some testings with various amount of data.

RedHat Linux 6.0
Kernel 2.2.5-smp
512MB RAM
Sort mem: 80MB
RAID5

100 million tuples 1:31
200 4:24
300 7:27
400 11:11 <-- 970MB
500 14:01 <-- 1.1GB (segmented files)
600 18:31
700 22:24
800 24:36
900 28:12
1000 32:14

I didn't see any bad thing at 1.1GB (500 million).
--
Tatsuo Ishii