Re: aio/README.md comments

Hi,

On 2025-08-29 15:23:48 -0700, Jeff Davis wrote:
> On Fri, 2025-08-29 at 12:32 -0400, Andres Freund wrote:
> > I don't really see an advantage of sync in those cases either.
>
> It seems a bit early to say that it's just there for debugging. But
> it's just in a README, so I won't argue the point.

There might be some regressions that make io_method=sync beneficial, but short
to medium term, the goal ought to be to make all non-ridiculous configurations
(I don't care about AIO performing well with s_b=16) to not regress
meaningfully and for most things to be the same or better with AIO.

I don't see any reason for io_method=sync to be something we should have for
anything other than debugging medium to long term.

Why do you think different?

> diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
> index 72ae3b3737d..8fa6bd6e9ca 100644
> --- a/src/backend/storage/aio/README.md
> +++ b/src/backend/storage/aio/README.md
> @@ -4,27 +4,38 @@
>
> ### Why Asynchronous IO
>
> -Until the introduction of asynchronous IO postgres relied on the operating
> -system to hide the cost of synchronous IO from postgres. While this worked
> -surprisingly well in a lot of workloads, it does not do as good a job on
> -prefetching and controlled writeback as we would like.
> -
> -There are important expensive operations like `fdatasync()` where the operating
> -system cannot hide the storage latency. This is particularly important for WAL
> -writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
> -writes can yield significantly higher throughput.

I think this second paragraph was important and your rewrite largely removed
it?

> +Postgres depends on IO operations happening asynchronously for reasonable
> +performance: for instance, a sequential scan would be far slower without the
> +benefit of readahead. Historically, Postgres only used synchronous APIs for
> +IO, while assuming that the operating system would use the kernel buffer cache
> +to make those operations asynchronous in most cases (aside from, e.g.,
> +`fdatasync()`).
> +
> +The asynchronous IO APIs described here do not depend on that
> +assumption. Instead, they allow different low-level IO methods, which are
> +given more control and therefore rely less on the kernel's
> +behavior. Currently, only async read operations are supported, but the
> +infrastructure is designed to support async write operations in the future.

The infrastructure supports writes today, it's just md.c and bufmgr.c isn't
aren't ready to use it today.

> ### Why Direct / unbuffered IO
>
> The main reasons to want to use Direct IO are:
>
> -- Lower CPU usage / higher throughput. Particularly on modern storage buffered
> - writes are bottlenecked by the operating system having to copy data from the
> - kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
> - can often move the data directly between the storage devices and postgres'
> - buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
> - perform other work.
> +- Avoid extra memory copies between the kernel buffer cache and Postgres
> + shared buffers. These memory copies can become the bottleneck when the
> + underlying storage has high enough throughput, which is common for
> + solid-state drives or fast network block devices. Instead, direct IO can
> + often move the data directly between the Postgres buffer cache and the
> + device by using DMA, leaving the CPU free to perform other work.
> - Reduced latency - Direct IO can have substantially lower latency than
> buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
> write latency.

I preferred the prior formulation that had the main reasons at the start of
the bullet points.

> @@ -37,11 +48,24 @@ The main reasons *not* to use Direct IO are:
>
> - Without AIO, Direct IO is unusably slow for most purposes.
> - Even with AIO, many parts of postgres need to be modified to perform
> - explicit prefetching.
> + explicit prefetching (see read_stream.c).
> - In situations where shared_buffers cannot be set appropriately large,
> e.g. because there are many different postgres instances hosted on shared
> hardware, performance will often be worse than when using buffered IO.

Ok, although perhaps better to refer to the read stream section at the bottom?

> +### Writing WAL
> +
> +Using AIO and Direct IO can reduce the overhead of WAL logging
> +substantially:
> +
> +- AIO allows to start WAL writes eagerly, so they complete before needing to
> + wait
> +- AIO allows to have multiple WAL flushes in progress at the same time
> +- Direct IO can reduce the number of roundtrips to storage on some OSs
> + and storage HW (buffered IO and direct IO without O_DSYNC needs to
> + issue a write and after the write's completion a cache flush,
> + whereas O\_DIRECT + O\_DSYNC can use a single Force Unit Access
> + (FUA) write).

> ## AIO Usage Example
>
> @@ -196,25 +220,15 @@ processing to the AIO workers).
>
> ### IO can be started in critical sections
>
> -Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
>
> -- AIO allows to start WAL writes eagerly, so they complete before needing to
> - wait
> -- AIO allows to have multiple WAL flushes in progress at the same time
> -- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
> - the number of roundtrips to storage on some OSs and storage HW (buffered IO
> - and direct IO without O_DSYNC needs to issue a write and after the write's
> - completion a cache flush, whereas O\_DIRECT + O\_DSYNC can use a single
> - Force Unit Access (FUA) write).

Direct IO alone does not reduce the number of roundtrips, the combination of
DIO and O_DSYNC does. I think that got less clear in the rewrite.

Greetings,

Andres Freund