Re: Compression and on-disk sorting

Jim C. Nasby wrote:
> On Mon, May 15, 2006 at 02:18:03PM -0400, Tom Lane wrote:
>
>> "Jim C. Nasby" <jnasby(at)pervasive(dot)com> writes:
>>
>>> A recent post Tom made in -bugs about how bad performance would be if we
>>> spilled after-commit triggers to disk got me thinking... There are
>>> several operations the database performs that potentially spill to disk.
>>> Given that any time that happens we end up caring much less about CPU
>>> usage and much more about disk IO, for any of these cases that use
>>> non-random access, compressing the data before sending it to disk would
>>> potentially be a sizeable win.
>>>
>> Note however that what the code thinks is a spill to disk and what
>> actually involves disk I/O are two different things. If you think
>> of it as a spill to kernel disk cache then the attraction is a lot
>> weaker...
>>
>
> I'm really starting to see why other databases want the OS out of their
> way...
>

Some of it is pure NIH syndrome. I recently heard of some tests done by
a major DB team that showed their finely crafted raw file system stuff
performing at best a few percent better than a standard file system, and
sometimes worse. I have often heard of the supposed benefits of our
being able to go behind the OS, but I am very dubious about it. What
makes people think that we could do any better than the OS guys?

cheers

andrew