Re: ice-broker scan thread

On Tue, Nov 29, 2005 at 02:53:36PM +1100, Gavin Sherry wrote:
> The second idea is using posix async IO at key points within the system
> to better parallelise CPU and IO work. There areas I think we could use
> async IO are: during sequential scans, use async IO to do pre-fetching of
> blocks; inside WAL, begin flushing WAL buffers to disk before we commit;
> and, inside the background writer/check point process, asynchronously
> write out pages and, potentially, asynchronously build new checkpoint segments.

I actually worked on this and got it to the stage where it wouldn't
crash anymore. It basically added a command to bufmgr.c called
PrefetchBuffer() which would initiate a request but not block. I then
hooked a few strategic places to call this. In particular during an
index scan, it would prefetch the next index block and the next few
data blocks and then return them in order as they came in.

Unfortunatly I can't really test it at it's full potential because it
uses glibc's default POSIX AIO which is *lame*. No more than one
outstanding request per fd which for PostgreSQL is crappy. There was
some evidence that in an index scan of a highly uncorrelated index that
it did make a small difference, but I never got around to testing it
fully. But bitmap scans already hugely reduce the cost of uncorrelated
indexes.

It doesn't pass regression because index_getmulti doesn't do backward
scans. Everything else works though.

If anyone is interested in the code I can send it to them. The results
on my system just wern't good enough to justify a lot more effort.

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.