Re: ice-broker scan thread

"David Boreham" <david_list(at)boreham(dot)org> wrote
>>
> I don't think your NT overlapped I/O code is quite right. At least
> I think it will issue reads at a high rate without waiting for any of them
> to complete. Beyond some point that has to give the kernel gut-rot.
>

[also with reply to Gavin] look up dictionary for "gut-rot", got it ... Uh,
this behavior is intended - I try to push enough requests shortly to kernel
so that it understands that I am doing sequential scan, so it would pull the
data from disk to file system cache more efficiently. Some file systems may
have "free-behind" mechanism, but our main thread (who really process the
query) should be fast enough before the data vanished.

>
> You could re-write your program to have a single thread but use aio.
> In that case it should show the same read ahead benefit that you see
> with the thread.
>

I guess this is also Gavin's point - I understand that will be two different
methodologies to handle "read-ahead". If no other thread/process involved,
then the main thread will be responsible to grab a free buffer page from
bufferpool and ask the kernel to put the data there by sync IO (current
PostgreSQL does) or async IOs. And that's what I want to avoid. I'd like to
use a dedicated thread/process to "break the ice" only, i.e., pull data from
disk to file system cache, so that the main thread will only issue *logical*
read.

Regards,
Qingqing