Re: About these IPC parameters

On Thu, 20 Jul 2000, Tom Lane wrote:

> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > We use three shared-memory segments: One is for the spin locks and is of
> > negligible size (144 bytes currently). The other two I don't know, but one
> > of them seems to be sized about 550kB + -B * BLCKSZ
>
> The shmem sizes depend on both -B and -N, but the dependence on -B is
> much stronger. Obviously there's 8K per -B for the buffer itself,
> and there's also some allowance for hashtable entries for the buffer
> indexing tables. The -N number drives the size of the PROC table plus
> some hashtables --- but a PROC entry isn't very big.
>
> I believe there's no really fundamental reason why we use three shmem
> segments and not just one. I've toyed with the idea of trying to
> combine them, but not done anything about it yet...
>
> > My kernel has the following interesting-looking shared memory settings:
>
> FWIW, HPUX does not have SHMALL --- and since HPUX began life as SysV
> I would imagine a lot of other SysV derivatives don't either. The
> relevant parameters here seem to be
>
> SEMA Enable Sys V Semaphores
> SEMAEM Max Value for Adjust on Exit Semaphores
> SEMMAP Max Number of Semaphore Map Entries
> SEMMNI Number of Semaphore Identifiers
> SEMMNS Max Number of Semaphores
> SEMMNU Number of Semaphore Undo Structures
> SEMUME Semaphore Undo Entries per Process
> SEMVMX Semaphore Maximum Value
> SHMEM Enable Sys V Shared Memory
> SHMMAX Max Shared Mem Segment (bytes)
> SHMMNI Number of Shared Memory Identifiers
> SHMSEG Shared Memory Segments per Process
>
> Other than shooting yourself in the foot by having SEMA or SHMEM be
> 0 (OFF), it looks like the parameters that could need raising on this
> platform would be SEMMAP, SEMMNI, SEMMNS, SHMMAX.
>
> > Is there any noteworthy relevance of some of the other parameters? I see
> > FAQ_BSDI talks about SEMUME and SEMMNU.
>
> AFAIK we don't use semaphore undo, so those are red herrings.

First off, this might be something we need a whole seperate FAQ for, since
I think the concepts are pretty much common across the various OSs?

for instance, under FreeBSD, I have it set right now as:

====
options SYSVSHM
options SHMMAXPGS=4096
options SHMSEG=256

options SYSVSEM
options SEMMNI=256
options SEMMNS=512
options SEMMNU=256
options SEMMAP=256

options SYSVMSG #SYSV-style message queues
====

To run three postmasters, one with '-B 256 -N 128', and the other two just
with '-N 16' ... the thing that I just don't get is how the settings ni my
kernel apply, and trying to find any info on taht is like pulling teeth :(
For instance, I'm allowing for up to 160 clients to connect, max .. does
that make for one semaphore identifier per client, so I need SEMMNI >=
160? Or ... ?

I grab'd this off a Sun site dealing with Solaris, but it might also be of
aid:

Name Default Max Brief Description
------ ------- -------------- -------------------------------------

semmap 10 2147483647 Number of entries in semaphore map

semmni 10 65535 Number of semaphore sets (identifiers)

semmns 60 2147483647 Number of semaphores in the system
65535 (usage)

semmnu 30 2147483647 Number of "undo" structures in the system

semmsl 25 2147483647 Max number of semaphores, per semaphore id
65535 (usage)

semopm 10 2147483647 Max number of operations, per semaphore call

semume 10 2147483647 Max number of "undo" entries, per process

semusz 96 *see below* Size in bytes of "undo" structure

semvmx 32767 2147483647 Semaphore maximum value
65535 (usage)

semaem 16384 2147483647 Adjust on exit maximum value
32767 (usage)

Detailed Descriptions
---------------------

semmap

Defines the size of the semaphore resource map; each block of available,
contiguous semaphores requires one entry in this map. This is the pool from
which semget(2) acquires semaphore sets.

When a semaphore set is removed (deleted), if the block of semaphores to be
freed is adjacent to a block of semaphores already in the resource map, the
semaphores in the set being removed are added to the existing map entry;
no new map entry is required. If the semaphores in the removed set are not
adjacent to those in an existing map entry, then a new map entry is required
to track these semaphores; if there are no more map entries available, the
system has to discard an entry, 'permanently' losing a block of semaphores
(permanence is relative; a reboot fixes the problem). If this should occur,
a WARNING will be generated, the text of which will be something like
"rmallocmap: rmap overflow, lost ...". The end result is that a user could
later get ENOSPC errors from semget(2) even though it doesn't look like all
the semaphores are allocated.

semmni

Defines the number of semaphore sets (identifiers), system wide. Every
semaphore set in the system has a unique indentifier and control structure.
The system pre-allocates kernel memory for semmni control structures; each
control structure is 84 bytes. If no more identifiers are available,
semget(2) returns ENOSPC.

Attempting to set semmni to a value greater than 65535 will result in
generation of a WARNING, and the value will be set to 65535.

semmns

Defines the number of semaphores in the system; 16 bytes of kernel memory is
pre-allocated for each semaphores. If there is not a large enough block of
contiguous semaphores in the resource map (see semmap) to satisfy the request,
semget(2) returns ENOSPC.

Fragmentation of the semaphore map will result in ENOSPC errors, even though
there may appear to be ample free semaphores. Despite attempts by the system
to merge free sets (see semmap), the size of the clusters of free semaphores
generally decreases over time. For this reason, semmns frequently must be set
higher than the actual number of semaphores required.

semmnu

Defines the number of semaphore undo structures in the system. semusz (see
below) bytes of kernel memory are pre-allocated for each undo structure; one
undo structure is required for every process for which undo information must
be recorded. semop() will return ENOSPC if it is requested to record undo
information and there are no undo structures available.

semmsl

Limits the number of semaphores that can be created for a single semaphore id.
If semget(2) returns EINVAL, this limit should be increased. This parameter
is only used to validate the argument passed to semget(2). Logically, it
should be less than or equal to semmns (see above). Setting semmsl too high
might allow a few identifiers to hog all the semaphores in the system.

semopm

Limits the number of operations that are allowed in a single semop(2) call.
If semop(2) returns E2BIG, this limit should be increased. This parameter is
only used to validate the argument passed to semop(2).

semume

Limits the number of undo records that can exist for a process. If semop(2)
returns EINVAL, this limit should be increased. In addition to its use in
validating arguments to semop(2), this parameter is used to calculate the
value of semusz (see below).

semusz

Defines the size of the semaphore undo structure. Any attempt to modify this
parameter directly will be ignored; semusz is always calculated based upon
the value of semume (see above); semusz = 8 * (semume + 2).

semvmx

Limits the maximum value of a semaphore. Due to the interaction with undo
structures and semaem (see below), this tuneable should not be increased
beyond its default value of 32767, unless you can guarantee that SEM_UNDO is
never and will never be used. It can be safely reduced, but doing so provides
no savings.

semaem

Limits the maximum value of an adjust-on-exit undo element. No system
resources are allocated based on this value.