Re: Recovery does not honor io_combine_limit, causing IOPS saturation

Hi Markos, Andrey,

I did some benchmarks and it was somewhat a disappointment, but I'll
summarize it here. There are some improvements, but not as much as I
expected. I leave it to you to decide if it is worth merging. If you
have suggestions for other measurements, I'm happy to do them.

Summary
--------------

Run on 2026-11-08 in `eu-north-1`, on a fresh `m6i.xlarge` with a fresh
50 GiB gp3 volume (3,000 IOPS, 125 MiB/s, XFS), the baseline commit
(unpatched) and the patched commit (read-ahead cache), built as release
(`-O3`, no assertions). 99 timed runs and 4 untimed `strace` count runs.
Cold unless stated otherwise.

Workload: `pgbench -i` bulk load, then `kill -9`. The WAL was generated
with the baseline binary and both binaries recover the same pristine
crashed copy.

Sizes: pgbench scales 25, 100 and 300, i.e. 0.30, 1.19 and 3.57 GiB of
WAL to replay.

Setup: one `m6i.xlarge`, `gp3` volume with 3,000 IOPS and 125 MiB/s, XFS,
crash recovery of 3.57 GiB of WAL (pgbench bulk load), 3 repetitions per
setting, cold page cache unless stated.

Different settings of `io_combine_limit` (cold cache)

| Setting           | Dev reads | pread64 | Redo s | CPU s |
| ----------------- | --------- | ------- | ------ | ----- |
| Baseline          | 59,921    | 468,121 | 53.5   | 5.82  |
| Patched, 16 pages | 58,779    | 29,408  | 53.2   | 5.76  |
| Patched, 32 pages | 44,168    | 14,784  | 50.1   | 5.46  |
| Patched, 64 pages | 44,156    | 7,472   | 49.4   | 5.18  |
| Baseline, ra=256  | 45,067    | n/c     | 50.3   | 5.42  |

Baseline = unpatched build, ra=256 = `read_ahead_kb=256` (default 128).
Dev reads = device read I/Os; CPU = startup process user+system. n/c =
not counted (expected to equal the baseline's, since readahead does not
change the calls).

* Device reads fall 25% at 32 pages. Redo time falls 5.8% (32 pages) and
  7.1% (64 pages) and startup CPU 5.3% and 10.1%, compared with the
  patch at its default of 16 pages.
* The unpatched baseline with read_ahead_kb=256 has 45,067 device reads
  (-24.8%), redo -6.0% and startup CPU -6.8%. The patch with 32 pages is
  within 2.0% of that on device reads and 0.3% on redo time. So on this
  setup (gp3, XFS, cold cache) `read_ahead_kb=256` gives the same read
  reduction without a code change.
* The patch still helps CPU at large limits: with 64 pages, startup CPU
  is -4.4% compared with the baseline at read_ahead_kb=256 (95% CI of
  the difference [-0.31, -0.16] s). This is probably the saved syscalls.
* Device reads stop falling at 256 KiB: 64 pages gives the same 44,156
  reads as 32 pages.
* Time until ready is 66.2 s in every setting. Recovery is limited by
  the volume's 125 MiB/s throughput.
* Average IOPS (reads + writes per second) was 1,008 to 1,253 in all
  settings, well under the volume's 3,000 limit, and neither the
  volume's nor the instance's IOPS limit was exceeded. So fewer reads
  did not speed recovery up here.

Sorry about the noise,
Mats Kindahl

On 9/14/26 20:44, Andrey Borodin wrote:
> Hi Mats, Markos,
>
> Mats, in the original gp3 setup, did you also measure fewer device-level
> reads and shorter recovery time with the normal readahead settings?
>
> I'd also be interested in replay performance with WAL and relation pages
> already cached. Fewer syscalls could help even when the kernel already
> combines disk reads, but does that saving outweigh the extra per-page
> memcpy in this implementation?
>
> Looking at v1, I think there is a cache-validity issue with streaming:
>
>> + if (readSource == XLOG_FROM_STREAM)
>> + wanted = Min(wanted, readLen);
>> ...
>> + wanted = Max(wanted, XLOG_BLCKSZ);
> readLen is the valid prefix of the requested page, not the amount of
> available WAL from that page onwards. It is at most XLOG_BLCKSZ, so
> these bounds also leave streaming reads at one page regardless of
> io_combine_limit.
>
> More importantly, when only part of the page has arrived, pread() still
> reads the whole page and readAheadLen records the full result. Once
> more WAL arrives, XLogPageRead() can be called again for the same page
> with a larger reqLen. The new cache then hits and returns the old copy,
> including bytes that were not valid when it was filled.
>
> Should the cache track how many bytes were valid at the time of the
> read, and refill when reqLen exceeds that prefix? The available WAL
> distance from targetPagePtr to flushedUpto could separately bound the
> combined read size.
>
> Thanks!
>
>
> Best regards, Andrey Borodin.