pgsql: Improve ReadRecentBuffer() scalability

Improve ReadRecentBuffer() scalability

While testing a new potential use for ReadRecentBuffer(), Andres
reported that it scales badly when called concurrently for the same
buffer by many backends. Instead of a naive (but wrong) coding with
PinBuffer(), it used the spinlock, so that it could be careful to pin
only if the buffer was valid and holding the expected block, to avoid
breaking invariants in eg GetVictimBuffer(). Unfortunately that made it
less scalable than PinBuffer(), which uses compare-exchange instead.

We can fix that by giving PinBuffer() a new skip_if_not_valid mode that
doesn't pin invalid buffers. It might occasionally skip when it
shouldn't due to the unlocked read of the header flags, but that's
unlikely and perfectly acceptable for an opportunistic optimisation
routine, and it can only succeed when it really should due to the
compare-exchange loop.

Note that this fixes ReadRecentBuffer()'s failure to bump the usage
count. While this could be seen as a bug, there currently aren't cases
affected by this in core, so it doesn't seem worth backpatching that portion.

Author: Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
Reported-by: Andres Freund <andres(at)anarazel(dot)de>
Reviewed-by: Andres Freund <andres(at)anarazel(dot)de>
Reviewed-by: Matthias van de Meent <boekewurm+postgres(at)gmail(dot)com>
Discussion: https://postgr.es/m/20230627020546.t6z4tntmj7wmjrfh%40awork3.anarazel.de
Discussion: https://postgr.es/m/fvfmkr5kk4nyex56ejgxj3uzi63isfxovp2biecb4bspbjrze7@az2pljabhnff

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/819dc118c0f6cd1fc08f0e807702b4bc0b0d8733

Modified Files
--------------
src/backend/storage/buffer/bufmgr.c | 64 ++++++++++++++++---------------------
1 file changed, 28 insertions(+), 36 deletions(-)