Re: Compression and on-disk sorting

On Fri, May 26, 2006 at 12:35:36PM -0400, Tom Lane wrote:
> "Jim C. Nasby" <jnasby(at)pervasive(dot)com> writes:
> > Something else worth mentioning is that sort performance is worse with
> > larger work_mem for all cases except the old HEAD, prior to the
> > tuplesort.c changes. It looks like whatever was done to fix that will
> > need to be adjusted/rethought pending the outcome of using compression.
>
> Please clarify. What are you comparing here exactly, and what cases did
> you test?

Sorry, forgot to put the url in:
http://jim.nasby.net/misc/pgsqlcompression/compress_sort.txt

But the meat is:
-- work_mem --
Scale 2000 20000
not compressed 150 805.7 797.7
not compressed 3000 17820 17436
compressed 150 371.4 400.1
compressed 3000 8152 8537
compressed, no headers 3000 7325 7876

Performance degrades with more work_mem any time compression is used. I
thought I had data on just your tuplesort.c change without compression,
but I guess I don't. :( I can run that tonight if desired.

As for the code, the 3 things I've tested are HEAD as of 5/17/06 with no
patches (labeld 'not compressed'); that code with the compression patch
(compressed), and that code with both the compression patch and your change to
tuplesort.c that removes tuple headers from the sorted data (compressed, no
headers).
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461