pgsql: bufmgr: Improve StartBufferIO interface

bufmgr: Improve StartBufferIO interface

Until now StartBufferIO() had a few weaknesses:

- As it did not submit staged IOs, it was not safe to call StartBufferIO()
where there was a potential for unsubmitted IO, which required
AsyncReadBuffers() to use a wrapper (ReadBuffersCanStartIO()) around
StartBufferIO().

- With nowait = true, the boolean return value did not allow to distinguish
between no IO being necessary and having to wait, which would lead
ReadBuffersCanStartIO() to unnecessarily submit staged IO.

- Several callers needed to handle both local and shared buffers, requiring
the caller to differentiate between StartBufferIO() and StartLocalBufferIO()

- In a future commit some callers of StartBufferIO() want the BufferDesc's
io_wref to be returned, to asynchronously wait for in-progress IO

- Indicating whether to wait with the nowait parameter was somewhat confusing
compared to a wait parameter

Address these issues as follows:

- StartBufferIO() is renamed to StartSharedBufferIO()

- A new StartBufferIO() is introduced that supports both shared and local
buffers

- The boolean return value has been replaced with an enum, indicating whether
the IO is already done, already in progress or that the buffer has been
readied for IO

- A new PgAioWaitRef * argument allows the caller to get the wait reference is
desired. All current callers pass NULL, a user of this will be introduced
subsequently

- Instead of the nowait argument there now is wait

This probably would not have been worthwhile on its own, but since all these
lines needed to be touched anyway...

Author: Andres Freund <andres(at)anarazel(dot)de>
Author: Melanie Plageman <melanieplageman(at)gmail(dot)com>
Discussion: https://postgr.es/m/zljergweqti7x67lg5ije2rzjusie37nslsnkjkkby4laqqbfw@3p3zu522yykv

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/74eafeab1a576620879e889ec70bf826e3e7fce8

Modified Files
--------------
src/backend/storage/buffer/bufmgr.c | 295 +++++++++++++++++-----------
src/backend/storage/buffer/localbuf.c | 46 +++--
src/include/storage/buf_internals.h | 24 ++-
src/test/modules/test_aio/t/001_aio.pl | 24 +--
src/test/modules/test_aio/test_aio--1.0.sql | 2 +-
src/test/modules/test_aio/test_aio.c | 17 +-
src/tools/pgindent/typedefs.list | 1 +
7 files changed, 259 insertions(+), 150 deletions(-)