RE: effective_io_concurrency and NVMe devices

Hi Nathan,

> > NVMe devices have a maximum queue length of 64k:
[..]
> > but our effective_io_concurrency maximum is 1,000:
[..]
> > Should we increase its maximum to 64k? Backpatched? (SATA has a
> > maximum queue length of 256.)
>
> If there are demonstrable improvements with higher values, this seems
> reasonable to me. I would even suggest removing the limit completely so
> this doesn't need to be revisited in the future.

Well, are there any? I remember playing with this (although for ANALYZE Stephen's case [1]) and got quite contrary results [2] -- see going to 16 from 8 actually degraded performance.
I somehow struggle to understand how 1000+ fadvise() syscalls would be a net benefit on storage with latency of ~0.1.. 0.3ms as each syscall on it's own is overhead (quite contrary, it should help on very slow one?)
Pardon if I'm wrong (I don't have time to lookup code now), but maybe Bitmap Scans/fadvise() logic would first need to perform some fadvise() offset/length aggregations to bigger fadvise() syscalls and in the end real hardware observable I/O concurrency would be bigger (assuming that fs/LVM/dm/mq layer would split that into more parallel IOs).

-J.

[1] - https://commitfest.postgresql.org/30/2799/
[2] - https://www.postgresql.org/message-id/flat/VI1PR0701MB69603A433348EDCF783C6ECBF6EF0(at)VI1PR0701MB6960(dot)eurprd07(dot)prod(dot)outlook(dot)com