Recovery does not honor io_combine_limit, causing IOPS saturation

Hi all,

I have been running recovery of Postgres clusters and small AWS
instances and
with a long history to replay, this quickly eats up the I/O budget for the
device. For example, with a gp3 with an IOPS limit of 3000 you quickly
eat it up
and get long stalls to recover the cluster. This is because the WAL
replay reads
the WAL in 8 KiB block rather than honoring io_combine_limit, which
typically
could allow reading 256 KiB in one go.

Since Postgres already have the option io_combine_limit to combine multiple
block reads into a single read, it is straightforward to honor it during
recovery using a simple cache (see attached patch). Since WAL replay is
mostly
from beginning to end, it is a simple way to reduce IOPS usage.

With this patch, and the default io_combine_limit of 16, I managed to
reduce the
number of I/O reads significantly:

|            | reads  | read_bytes  | avg bytes/read |
|------------|--------|-------------|----------------|
| WAL before | 31,283 | 256,270,336 | 8,192          |
| WAL after  | 1,955  | 256,131,072 | 131,013        |

Some back-of-the-envelope calculations give that with this patch we
should not be
able to saturate the device interface with the maximum read bandwidth
for gp3 is
approx 125 MiB/s, but the maximum read speed if you want to not hit the
limit
when you read 8 KiB blocks is 23-24 MiB/s.

I thought I'd share the patch and the results and hear if people think
this is a
useful improvement. There is already some work on pre-fetching, but this
is such
a simple patch that I think it could be worth to add to the current version.

(I also have some Perl scripts to set up crash recovery and some bpftrace
scripts that checks the reads, if anybody is interested, but I need to clean
them up a little.)

Best wishes,
Mats Kindahl, Multigres Engineer, Supabase