Re: 9.1.3 backends getting stuck in 'startup'

> I think you can probably skip all that are blocked in LWLockAcquire
> called from bufmgr.c:531, at least for a first pass. Calls from
> elsewhere in bufmgr.c might be more interesting, and anything that's not
> blocked at an LWLockAcquire at all might be even more interesting.

A few more that include bufmgr.c:

#0 0x00007f62b6150583 in __select_nocancel () at
../sysdeps/unix/syscall-template.S:82
#1 0x00007f62b82948ca in pg_usleep (microsec=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/port/pgsleep.c:43
#2 0x00007f62b8159678 in CheckpointWriteDelay (flags=<value optimized out>,
progress=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/bgwriter.c:722
#3 0x00007f62b81806ab in BufferSync (flags=64) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/storage/buffer/bufmgr.c:1283
#4 CheckPointBuffers (flags=64) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/storage/buffer/bufmgr.c:1764
#5 0x00007f62b7fea0e3 in CheckPointGuts (checkPointRedo=..., flags=64) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/transam/xlog.c:7989
#6 0x00007f62b7ff5a3c in CreateCheckPoint (flags=64) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/transam/xlog.c:7797
#7 0x00007f62b8159a46 in BackgroundWriterMain () at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/bgwriter.c:502
#8 0x00007f62b80115c2 in AuxiliaryProcessMain (argc=<value optimized out>,
argv=0x7fff9045db00) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/bootstrap/bootstrap.c:422
#9 0x00007f62b815fe8b in StartChildProcess (type=BgWriterProcess) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:4504
#10 0x00007f62b8162ec2 in reaper (postgres_signal_arg=<value optimized out>)
at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:2424
#11 <signal handler called>
#12 0x00007f62b6150583 in __select_nocancel () at
../sysdeps/unix/syscall-template.S:82
#13 0x00007f62b81614d3 in ServerLoop () at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:1415
#14 0x00007f62b816457c in PostmasterMain (argc=-1186270896,
argv=0x7f62b9471190) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:1116
#15 0x00007f62b8102ec3 in main (argc=5, argv=0x7f62b9471170) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/main/main.c:199

#0 0x00007f62b6149a50 in __read_nocancel () at
../sysdeps/unix/syscall-template.S:82
#1 0x00007f62b8183ced in read (file=<value optimized out>,
buffer=0x7f613ff51fa0 ":\n", amount=<value optimized out>) at
/usr/include/bits/unistd.h:45
#2 FileRead (file=<value optimized out>, buffer=0x7f613ff51fa0 ":\n",
amount=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/storage/file/fd.c:1191
#3 0x00007f62b819c7a6 in mdread (reln=<value optimized out>, forknum=<value
optimized out>, blocknum=39946, buffer=0x7f613ff51fa0 ":\n") at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/storage/smgr/md.c:640
#4 0x00007f62b81807f1 in ReadBuffer_common (smgr=0x7f62b955d090,
relpersistence=<value optimized out>, forkNum=<value optimized out>,
blockNum=<value optimized out>, mode=<value optimized out>, strategy=<value
optimized out>, hit=0x7fff9045d5cf "") at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/storage/buffer/bufmgr.c:438
#5 0x00007f62b81811e4 in ReadBufferExtended (reln=0x7f62b7eef540,
forkNum=MAIN_FORKNUM, blockNum=39946, mode=RBM_NORMAL, strategy=0x4ff74188) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/storage/buffer/bufmgr.c:246
#6 0x00007f62b7fc97bf in heapgetpage (scan=0x7f62b9558350, page=39946) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/heap/heapam.c:223
#7 0x00007f62b7fca1ba in heapgettup (scan=0x7f62b9558350, dir=<value
optimized out>, nkeys=<value optimized out>, key=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/heap/heapam.c:556
#8 0x00007f62b7fca7bc in heap_getnext (scan=0x7f62b9558350,
direction=1073029024) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/heap/heapam.c:1345
#9 0x00007f62b8259570 in RelationBuildTupleDesc (targetRelId=<value optimized
out>, insertIt=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:468
#10 RelationBuildDesc (targetRelId=<value optimized out>, insertIt=<value
optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:882
#11 0x00007f62b825af26 in RelationIdGetRelation (relationId=2601) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:1568
#12 0x00007f62b7fcdaab in relation_open (relationId=2601, lockmode=1073029024)
at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/heap/heapam.c:907
#13 0x00007f62b7fcdb33 in heap_open (relationId=8, lockmode=1073029024) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/heap/heapam.c:1077
#14 0x00007f62b8252d82 in SearchCatCache (cache=0x7f62b94c9e70, v1=403,
v2=<value optimized out>, v3=0, v4=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/catcache.c:1184
#15 0x00007f62b8258a30 in RelationInitIndexAccessInfo
(relation=0x7f62b7efa828) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:1010
#16 0x00007f62b8259d91 in RelationBuildDesc (targetRelId=<value optimized
out>, insertIt=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:904
#17 0x00007f62b825a01d in load_critical_index (indexoid=2655, heapoid=2603) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:3080
#18 0x00007f62b825a4c3 in RelationCacheInitializePhase3 () at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:2890
#19 0x00007f62b82713e9 in InitPostgres (in_dbname=<value optimized out>,
dboid=<value optimized out>, username=<value optimized out>, out_dbname=<value
optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/init/postinit.c:816
#20 0x00007f62b81a2908 in PostgresMain (argc=-1206253552, argv=<value
optimized out>, username=0x7f62b9471c90 "consprod") at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/tcop/postgres.c:3650
#21 0x00007f62b8161ad3 in BackendRun () at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:3606
#22 BackendStartup () at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:3291
#23 ServerLoop () at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:1455
#24 0x00007f62b816457c in PostmasterMain (argc=-1186270896,
argv=0x7f62b9471190) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:1116
#25 0x00007f62b8102ec3 in main (argc=5, argv=0x7f62b9471170) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/main/main.c:199

and I've got 81 more that do not contain bufmgr.c and are also not block on
LWLockAcquire.