Moving _bt_readpage and _bt_checkkeys into a new .c file

Attached patch v1-0001-* moves _bt_readpage (from nbtsearch.c) and
_bt_checkkeys (from nbtutils.c) into a new .c file -- nbtreadpage.c.
It also moves all of the functions that _bt_checkkeys itself calls
(either directly or indirectly) over to nbtreadpage.c. This first
patch is strictly mechanical, in that it only moves existing functions
around, without directly changing anything.

There's also a second patch, which adds a performance optimization
that builds on the first patch. This makes the functions moved to the
new file pass around the scan's direction, rather than requiring the
functions to fish it out of so->currPos as before.

With large range query variants of pgbench consisting of queries like
"SELECT abalance FROM pgbench_accounts WHERE aid between 1000 AND
3000", at scale 100, with a single client, I find that there's a ~4.5%
increase in transaction throughput with both patches applied (when
compared to master). The new module structure coupled with the
optimization in the second patch gives the compiler the ability to
produce more efficient code.

Here's a set of results from a series on interlaced 5 minute runs:

master: tps = 2682.600100, lat avg = 0.373 ms, lat stddev = 0.006 ms
cpu/txn = 0.011320, cs/txn = 8.1114, in/txn = 0.1832
patch: tps = 2827.020776, lat avg = 0.354 ms, lat stddev = 0.006 ms
cpu/txn = 0.010777, cs/txn = 8.1083, in/txn = 0.1768
master: tps = 2711.945605, lat avg = 0.369 ms, lat stddev = 0.006 ms
cpu/txn = 0.011173, cs/txn = 8.1109, in/txn = 0.1815
patch: tps = 2832.105849, lat avg = 0.353 ms, lat stddev = 0.006 ms
cpu/txn = 0.010793, cs/txn = 8.1090, in/txn = 0.1769
master: tps = 2699.348706, lat avg = 0.370 ms, lat stddev = 0.007 ms
cpu/txn = 0.011225, cs/txn = 8.1123, in/txn = 0.1813
patch: tps = 2822.511539, lat avg = 0.354 ms, lat stddev = 0.006 ms
cpu/txn = 0.010759, cs/txn = 8.1078, in/txn = 0.1760
master: tps = 2715.051289, lat avg = 0.368 ms, lat stddev = 0.006 ms
cpu/txn = 0.011160, cs/txn = 8.1116, in/txn = 0.1824
Patch: tps = 2818.002045, lat avg = 0.355 ms, lat stddev = 0.007 ms
cpu/txn = 0.010752, cs/txn = 8.1078, in/txn = 0.1762
master: tps = 2720.488826, lat avg = 0.367 ms, lat stddev = 0.011 ms
cpu/txn = 0.011138, cs/txn = 8.1103, in/txn = 0.1815
Patch: tps = 2818.035923, lat avg = 0.355 ms, lat stddev = 0.006 ms
cpu/txn = 0.010752, cs/txn = 8.1087, in/txn = 0.1762

Other kinds of queries benefit much less. I think that there's a sub
1% improvement with standard pgbench SELECT, though it's hard to
distinguish from noise. There are fewer code paths affected by the
second patch in use with such queries, compared to the favorable range
scan query case that I've shown detailed numbers for, which might
explain why simple equality lookups aren't improved by very much.

This seems like an enhancement that is pretty easy to justify. Note
that the changes in the second patch essentially restore things to how
they already were prior to my commit 763d65ae. I doubt that that
change caused a regression at the time, since the speedup that I see
now depends on the changes in the first patch (though I must admit
that I haven't benchmarked the changes made by the second patch in
isolation).

--
Peter Geoghegan