Re: [HACKERS] sort on huge table

From: Tatsuo Ishii <t-ishii(at)sra(dot)co(dot)jp>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Don Baccus <dhogaza(at)pacifier(dot)com>, Mike Mascari <mascarim(at)yahoo(dot)com>, t-ishii(at)sra(dot)co(dot)jp, pgsql-hackers(at)postgreSQL(dot)org
Subject: Re: [HACKERS] sort on huge table
Date: 1999-11-02 04:07:32
Message-ID: 199911020407.NAA01247@srapc451.sra.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

>> At 06:24 PM 11/1/99 -0800, Mike Mascari wrote:
>>> I know this is a VERY long shot, but... what were the READ/WRITE ratios
>>> between the old version and the new version? Perhaps the computation
>>> of the checksum (sic) blocks under RAID5 caused the unexpected behavior.
>
>Good try but no cigar --- we're dealing with a merge algorithm here,
>and it's inherently the same amount of data in and out. You write
>a block once, you read the same block once later on. But...
>
>Don Baccus <dhogaza(at)pacifier(dot)com> writes:
>> RAID 5, not the operating system, might be getting in the way...it
>> would be interesting to test this on a Linux 2.2 kernel without
>> the RAID 5 complication.
>
>... I agree this'd be worth trying. There could be some subtle effect
>somewhere in RAID5 that's tripping things up. It'd also be useful if
>someone could try it on similar RAID hardware with a non-Linux kernel.

I have compared current with 6.5 using 1000000 tuple-table (243MB) (I
wanted to try 2GB+ table but 6.5 does not work in this case). The
result was strange in that current is *faster* than 6.5!

RAID5
current 2:29
6.5.2 3:15

non-RAID
current 1:50
6.5.2 2:13

Seems my previous testing was done in wrong way or the behavior of
sorting might be different if the table size is changed?

Anyway, here is my test script.
First, edit Makefile to set DB and number of tuples. Then run type
make. That's all.
--
Tatsuo Ishii
--------------------------------------------------------------------
begin 644 sort.tar.gz
M'XL(`(-F'C(at)``^V737/:,!"&N5:_8(at)MD@@D8VQ(at)\`R$S!=)V.J3I).TIR<'8
M,H@:F](at)BDTQ+?WNU_H#2-L(dot)ED![T<$`?[Z[6R"LM<1CQQH7]E7K,IX7]H&M:
MVS2A`-!LZDW\UJV6AM\I+5,#L#3-,LVV9NIBVA"?`FA[BF>+9<SM"*#`ZRR>
M,O:L;CZ)#A'(dot)H2GM(at)I3`I9Z]]#G$E',63&+PP@@XC;&CDF$?>DF/?/S\Y=/H
M_%ITQ8XC9+=W,A@(_<1QR.#MZ,T[-*Y?&N3Z_?EH)-J-,0L:\91<7>2=:`YU
MCUQ?#5`ZH8'JD,O^A[P3$F+[?@>B99"$1+)&!V?)JW(E<:R`VG`B:G.JQE,H
M5[+`%=$<]A5"A+8CVNA70:/!`*?2`+&%JRM0#Q.GQ/&I'710=W6AX%!N"M4?
MY*7W=Q<QYG_R,^YOC9WYWVZE^6\8>E/'_->;EBGS_Q"46.#X2Y?":<Q=%JK3
M,T+F-(at)LJ+(dot)!@1Q(dot)G!LY4_$#5JN(at)\*(dot)0;`<`I5IMULV:0)WR7X(@'%;2#,]`5
M0#DD"IN'#"<>;O0[!4U7J/86D7#A5<3B-(IJ1=P,/%;@R`4G7#P5:T$B1JDX
M="JLIW79:=!E)R>Y\\Q#\<B]Y<4:2^29>B;4LU-#1#;;Z#<6C\5,O-IR=!OD
MX_(at)L["AY-NCU0-NX^#UN]5=7J[\\F!L&5$T]K_ZG0R')__PLO/?WLL:N_(>6
ME>6_WFZU+;S_35.3^7\(W"A<`+?'/@6N=TGZ)JP'*(at)PSO`9<7/"/7+R[+QVN
MY!^SE?_3_:RQ*_]-2\_J_Z:E-S7,?Z-MR/P_!*77ZQI;W'8W4"Y!/:!(at)P%V7
M3T5UB_LS[/>28AK;6:G<2RY%0OV8;@V7]=RB;!"/$;(0EXHHV&%SQ4!YV"=J
M(ZF3,[/OJ<J!8[SSQ;D#7A3.`2N2X#C14V<:0K&'P(/M+)=S2#I%DO;<,=3K
M^:GUA\6ZJKA?TN(at)IM^1L3F&]<DQ]ZG"HIDL+)V$D;FX8/P$3?X!B!WPV9QP+
FG32BE]XXB40BD4(at)D$HE$(I%()!*)1"*12"229_@)GWZ*5``H````
`
end

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Lamar Owen 1999-11-02 04:30:35 Re: [HACKERS] change in name of perl?
Previous Message Bruce Momjian 1999-11-02 04:05:47 6.5.3 is ready