Don't close relation files when the visibility map is extended

Greetings,

When vm_extend() grows the visibility map it sends an smgr
invalidation. Every backend that has the relation open then closes
all of its files, and reopens each segment the next time it touches
the table. VACUUM extends the map about once per 255MB of heap. On a
multi-TB table with a few hundred backends inserting, that is a lot of
opens, and as far as I can tell none of them are needed. The other
backends only have to learn that the map got bigger.

This assumes open() is cheap, and it isn't always. I ran into it on
17 with a ~4TB partition, on a system where open() goes through a
filesystem encryption layer. Each time VACUUM extended the map, every
inserting backend reopened about 4,000 segments, and inserts stalled
for seconds at a time. The same invalidation is in every branch from
14 to master, and on master on-access pruning can extend the map as
well, so it isn't only VACUUM.

The attached patch adds an invalidation message that just resets the
cached fork sizes, and vm_extend() sends that instead. Readers see
new map pages when they do now. I also tried dropping the
invalidation entirely, but then a backend that doesn't prune the
pages itself keeps a stale map size until VACUUM finishes.

On eCryptfs, with about 2,700 segments and 300 inserting clients,
VACUUM went from about 10 million opens to under 1 million, the extra
system CPU it caused mostly went away, and p99 insert latency during
VACUUM dropped from ~350ms to under 20ms. check-world passes.

Is there a reason the full smgr invalidation is needed here that I'm
missing?

--
Bryan Green
EDB: https://www.enterprisedb.com