[bug report] The backend process cannot reuse VfdCache cache entries

Hi All，

Recently, in a production environment, we encountered the following SQL error:  
`ERROR: dn_6001_6002: dn_6011_6012: invalid memory alloc request size 1174405120 in fd.cpp:1094`  
Based on this error message, the database was trying to allocate a VfdCache of size 1174405120 bytes, which exceeds the maximum allowable memory space `maxCacheSize` for a single allocation.

My analysis: 
- The error occurred at the moment VfdCache was being expanded. According to the VfdCache expansion strategy, we can deduce that the system had approximately 9.4 million files open at that moment (or 9.4 million cache entries allocated). From the error size and :
  
  newCacheSize = 1174405120 = 10485760 * 112
 
- At the time of the error, the system needed to expand VfdCache to 10,485,760 VFD entries, which must be the result of expansion strategy 2.

- The original VfdCache size `SizeVfdCache` would be: `10485760 - 1048576 = 9437184`, meaning there were already over 9.4 million VFD entries present before expansion.

## Why are there so many VFD entries?
**Root cause:**  
The free-list pointer of VfdCache does not reuse old slots. Whenever a session opens a new file, it occupies a new slot. `VfdCache[0].nextFree` never goes backwards. It keeps moving forward. After opening a certain number of files, it repeatedly triggers VfdCache expansion, and when the size exceeds the threshold, an error is reported.

## GDB debugging information
```sql
# Prepare three tables
postgres=# select oid, relfilenode from pg_class where relname in ('t1','t2','t3');
  oid  | relfilenode 
-------+-------------
 16384 |       16384
 24576 |       24576
 24579 |       24579
(3 rows)

# Execute the following three SQL statements sequentially in the same session
postgres=# select * from t1;
postgres=# select * from t2;
postgres=# select * from t3;

# After first sql executed , VFD cache information at transaction commit
(gdb) p VfdCache[0]
(gdb) p VfdCache[0]
$34 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 4, lruMoreRecently = 3, lruLessRecently = 3, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[1]
$35 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 2, lruMoreRecently = 2, lruLessRecently = 0, fileSize = 0, fileName = 0x31d9ec0 "base/5/2601", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[2]
$36 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 3, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x31da120 "base/5/16384", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[3]
$37 = {fd = 7, fdstate = 0, resowner = 0x0, nextFree = 4, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x31da140 "base/5/24576", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[4]
$38 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 5, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[5]
$39 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 6, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[6]
$40 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 7, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[7]
$41 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 8, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}

# After second sql executed , VFD cache information at transaction commit

(gdb) p VfdCache[0]
$34 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 4, lruMoreRecently = 3, lruLessRecently = 3, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[1]
$35 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 2, lruMoreRecently = 2, lruLessRecently = 0, fileSize = 0, fileName = 0x31d9ec0 "base/5/2601", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[2]
$36 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 3, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x31da120 "base/5/16384", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[3]
$37 = {fd = 7, fdstate = 0, resowner = 0x0, nextFree = 4, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x31da140 "base/5/24576", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[4]
$38 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 5, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[5]
$39 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 6, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[6]
$40 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 7, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[7]
$41 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 8, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}

# After third sql executed , VFD cache information at transaction commit

(gdb) p VfdCache[0]
$67 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 7, lruMoreRecently = 4, lruLessRecently = 5, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[1]
$68 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 2, lruMoreRecently = 2, lruLessRecently = 0, fileSize = 0, fileName = 0x31d9ec0 "base/5/2601", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[2]
$69 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 3, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x31da120 "base/5/16384", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[3]
$70 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 4, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x31da140 "base/5/24576", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[4]
$71 = {fd = 7, fdstate = 0, resowner = 0x0, nextFree = 5, lruMoreRecently = 6, lruLessRecently = 0, fileSize = 0, fileName = 0x31da160 "base/5/1249", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[5]
$72 = {fd = 9, fdstate = 0, resowner = 0x0, nextFree = 6, lruMoreRecently = 0, lruLessRecently = 6, fileSize = 0, fileName = 0x31da180 "base/5/24579", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[6]
$73 = {fd = 14, fdstate = 0, resowner = 0x0, nextFree = 7, lruMoreRecently = 5, lruLessRecently = 4, fileSize = 0, fileName = 0x31dc5e0 "base/5/2619", fileFlags = 524290, fileMode = 384}
(gdb) p VfdCache[7]
$74 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 8, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[8]
$75 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 9, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[9]
$76 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 10, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}
(gdb) p VfdCache[10]
$77 = {fd = -1, fdstate = 0, resowner = 0x0, nextFree = 11, lruMoreRecently = 0, lruLessRecently = 0, fileSize = 0, fileName = 0x0, fileFlags = 0, fileMode = 0}

## Analysis
1. When executing the first SQL statement `select * from t1;`, the physical file for table t1 (`filenode = 16384`) is stored in slot `VfdCache[2]`. At this point, the head pointer `VfdCache[0]` points to a free list where `VfdCache[0]->nextFree = 3`, meaning slots starting from 3 are free.  
   When executing the second SQL `select * from t2;`, the physical file for t2 (`filenode = 24576`) is stored in slot `VfdCache[3]`. Now `VfdCache[0]->nextFree = 4`, slots from 4 onward are free.  
   When executing the third SQL `select * from t3;`, the physical file for t3 (`filenode = 24579`) is stored in slot `VfdCache[5]` (this SQL opens 3 new files). The free list pointer becomes `VfdCache[0]->nextFree = 7`, slots from 7 onward are free.

2. During the third SQL, the file descriptors opened by previous SQL statements have been released: the physical file descriptor `VfdCache->fd` is closed and set to -1, but the VfdCache slots themselves are not released. When opening new files, the `nextFree` pointer always moves forward to new slots, and already-used slots are never reused by new files.  
   As a result, as long as the session connection remains open, VfdCache only grows. When the cumulative number of opened files becomes large enough, the VfdCache capacity exceeds the memory limit, causing SQL execution to fail.

3. Old VFD slots are bound to the SMGR and stored in the relation structure, which is cached in the relcache. If a subsequent SQL statement opens the same file again, it will reuse the cached VFD slot from the relcache.

4. VfdCache is at the backend process level. When the session disconnects, VfdCache is cleared. But if the connection stays open, the cache size keeps growing.

## Solutions
1. On the application side, use short-lived connections. Long-lived connections are more prone to this error.
2. Solve the problem from the database kernel side. I would like to ask the experts: is this a defect in VfdCache? If so, my idea is to make `VfdCache[0]->nextFree` point backwards when a SQL statement ends, so that the same connection can reuse old entries when opening new files.

That concludes my question. Looking forward to your reply.

BG

dengkai
784523565(at)qq(dot)com