Re: Don't close relation files when the visibility map is extended

On Mon, Oct 26, 2026 at 11:06 PM Bryan Green <dbryan(dot)green(at)gmail(dot)com> wrote:
>
> Greetings,
>
> When vm_extend() grows the visibility map it sends an smgr
> invalidation. Every backend that has the relation open then closes
> all of its files, and reopens each segment the next time it touches
> the table. VACUUM extends the map about once per 255MB of heap. On a
> multi-TB table with a few hundred backends inserting, that is a lot of
> opens, and as far as I can tell none of them are needed. The other
> backends only have to learn that the map got bigger.
>
> This assumes open() is cheap, and it isn't always. I ran into it on
> 17 with a ~4TB partition, on a system where open() goes through a
> filesystem encryption layer. Each time VACUUM extended the map, every
> inserting backend reopened about 4,000 segments, and inserts stalled
> for seconds at a time. The same invalidation is in every branch from
> 14 to master, and on master on-access pruning can extend the map as
> well, so it isn't only VACUUM.
>
> The attached patch adds an invalidation message that just resets the
> cached fork sizes, and vm_extend() sends that instead. Readers see
> new map pages when they do now. I also tried dropping the
> invalidation entirely, but then a backend that doesn't prune the
> pages itself keeps a stale map size until VACUUM finishes.
>
> On eCryptfs, with about 2,700 segments and 300 inserting clients,
> VACUUM went from about 10 million opens to under 1 million, the extra
> system CPU it caused mostly went away, and p99 insert latency during
> VACUUM dropped from ~350ms to under 20ms. check-world passes.
>
> Is there a reason the full smgr invalidation is needed here that I'm
> missing?

I've digged a little into histoy and got this context:

It appers that years earlier (see b4e074172768) smgrinvals were sent only for
truncation earlier. It also seems that code before than that ended up doing
lots of lseek?/open() without VM and this commit (optimization) assumed
(somewhat correctly) that open()/close() is cheap, but it's only has low
latency on standard stuff (ext4/XFS), but there might some (encrypted)
filesystems where that is not that cheap.

I've tracked down this commit down to this thread [1] which contains that
this was an optmization to a regression due to lack of VM in the first
place.

Anyway, we can repro extreme high 50%+ sys% (kernel) with with encFS(LUKS)
just with lots of close() /open() stress test, so avoiding this would help
users of such encrypted filesystems. E.g. pure open/close syscall benchmark
shows:
ext4 throughput 1736656 opens/s @ 8 processes
encfs throughput 87017 opens/s @ 8 processes
(of course it's FUSE/userland)but that's like ~20x slower and when hundreths
of such backends do receive such sinval (by VACCUUM -> vm_extend()) almost
all at once, it's almost DoS to the whole system shortly, the bigger the
impact the more segments (bigger relation/more files are being opened()/
closed()) multipled by backends handling sinvals. I think we can say about
that for three such filesystems already on Linux (two via FUSE, and at least
one implemented via LKM).

So question is: should we optimize further for such high-latency and costly
open()/close() filesystems? My thinking is that we could help vm_extend() here
avoiding doing that, but then there are other sinvals (e.g. due to DDLs) that
will also cause such "kernel stress" (hiccups).

BTW: @Bryan: I've created commitfest entry [2]

-J.

[1] - https://www.postgresql.org/message-id/flat/CAEZATCVufYwtULWuL5Um1YvnW1N8jMyBy%2BqkPU9Q1xOqCcMa-Q%40mail.gmail.com
[2] - https://commitfest.postgresql.org/patch/7528/