Re: Skip prefetch for block references that follow a FPW or WILL_INIT of the same block

On 10/27/26 06:58, Jeremy Schneider wrote:
>
>
> On Mon, Oct 26, 2026 at 8:52 PM shihao zhong <zhong950419(at)gmail(dot)com
> <mailto:zhong950419(at)gmail(dot)com>> wrote:
>
> I did not find a 16x amplification diagram in your github link.
> Maybe I am missing
> something?
>
>
> The IOPS amplification was very clear from looking at the charts showing
> Azure IOPS for the primary on the replica. But you're right that the
> github link did not show the math. The 16x comes from the primary doing
> 128k reads for seqscan and vacuum (whether due to io_uring or os
> readahead) while the replica does 8k reads (i'm fairly confident linux
> wont do readahead).  But when I visually look at the charts, it actually
> seems a bit higher than 16x - the replica was pegged at 500 IOPS for
> over 30 minutes.  I'm doing a bit more testing now to see if I can
> illustrate this more clearly.
>

I believe you're right posix_fadvise(WILLNEED) effectively disables
kernel readhead - which is unfortunate, of course. I haven't looked at
the kernel code recently, but an easy way to verify that is with a
simple C program that issues fadvise + read on a file.

I'm attaching an example I just mashed together (ugly, likely dangerous)
that gets a preallocated file, and issues sequential fadvise+read calls
on it, in chunks of 16 x 8K blocks. Something like

--
gcc -o fadvise-readahead fadvise-readahead.c

dd if=/dev/random bs=1M of=test.file count=8192 status=progress

sync
echo 3 > /proc/sys/vm/drop_caches

./fadvise-readahead /mnt/data/test.file
--

With the fadvise calls in place, iostat -x -k says something like

Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz
nvme5n1 56423.00 451384.00 0.00 0.00 0.42 8.00

while with the fadvise calls commented out, it says

Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz
nvme5n1 14142.00 1808736.00 0.00 0.00 0.21 127.90

So the average request size goes from 8K to 128K, which is exactly what
readahead does. But only without the fadvise stuff.

This only matters if there actually is sequential pattern. I assumed
it'd not make a significant difference without the IOPS limit, due to
readahead built into the SSDs, but it seems I was wrong (per the iostat
output above). On a random access it's a clear win, ofc.

I don't have a great solution to this. We'd need to do I/O combining,
similarly to what happens on the primary. I'm not sure if it's feasible,
or if it'd be better to rework the prefetcher to use read_stream (which
has all of this built in).

Sure, disabling prefetching is an option. But I'd point out the
prefetching was conveived as a solution to issues with ever-growing
replication lag. So it's a trade off.

regards

--
Tomas Vondra