Re: Linux max on shared buffers?

Curt Sampson <cjs(at)cynic(dot)net> writes:
> ironic $ /usr/pkg/bin/lmbench/lat_syscall null
> Simple syscall: 0.2026 microseconds

> ironic $ /usr/pkg/bin/lmbench/bw_mem 32m bcopy
> 33.55 352.69

> 352.69 MB/sec works out to about 370 bytes per microsecond. Thus,
> copying an 8K page should take a bit over 22 microseconds, enough
> time for the overhead of 110 syscalls.

> ironic $ /usr/pkg/bin/lmbench/bw_mem 64k bcopy
> 0.065536 2038.58

> Even entirely in L1 cache, I still get only about 2138 bytes per
> microsecond, thus taking almost 4 microseconds to copy a page, the same
> as the overhead for 8 syscalls.

Hm. What's the particular syscall being used for reference here?
And how does it compare to the sorts of activities we'd actually be
concerned about (open, close, mmap)?

> All this aside, remember that when you copy a block under the
> current system, you're doing it with read() or write(), and thus
> paying the syscall overhead anyway.

Certainly. But as you just demonstrated, the overhead of entering the
kernel is negligible compared to doing the copying (to say nothing of
any actual I/O required). I'm not convinced that futzing with a
process' memory mapping tables is free, however ... especially not if
you're creating a large number of separate small mappings. If mmap
provokes a TLB flush for your process, it's going to be expensive
(just how expensive will be hard to measure, too, since most of the
cycles will be expended after returning from mmap).

regards, tom lane