Re: Linux/PostgreSQL scalability issue - problem with 8 cores

Hi Tom & all,

>> It sounds a bit like momentary contention for a spinlock,
>> but exactly what isn't clear.

> ok, we're going to try oprofile, will let you know...

yes, it seems like contention for spinlock if I'm intepreting oprofile
correctly, around 60% of time during spikes is in s_lock. [for details
see below].

We've tried several times to get stacktrace from some of the running
backends during spikes, we got always this:

0x00002b005d00a9a9 in semop () from /lib/libc.so.6
#0 0x00002b005d00a9a9 in semop () from /lib/libc.so.6
#1 0x000000000054fe53 in PGSemaphoreLock (sema=0x2b00a04e5090,
interruptOK=0 '\0') at pg_sema.c:411
#2 0x0000000000575d95 in LWLockAcquire (lockid=SInvalLock,
mode=LW_EXCLUSIVE) at lwlock.c:455
#3 0x000000000056fbfe in ReceiveSharedInvalidMessages
(invalFunction=0x5e9a30 <LocalExecuteInvalidationMessage>,
resetFunction=0x5e9df0 <InvalidateSystemCaches>) at sinval.c:159
#4 0x0000000000463505 in StartTransactionCommand () at xact.c:1439
#5 0x000000000056fa4b in ProcessCatchupEvent () at sinval.c:347
#6 0x000000000056fb20 in CatchupInterruptHandler
(postgres_signal_arg=<value optimized out>) at sinval.c:221
#7 0x00002b005cf6f110 in killpg () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()

Is this enough info to guess what's happening? What should we try next?

Thanks,

Kuba

oprofile results:

[I've shortened path from /usr/local/pg... to just pg for better
readilibity]

# opreport --long-filenames
CPU: Core 2, speed 2666.76 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
unit mask of 0x00 (Unhalted core cycles) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
125577 90.7584 pg-8.2.4/bin/postgres
3792 2.7406 /lib/libc-2.3.6.so
3220 2.3272 /usr/src/linux-2.6.22.15/vmlinux
2145 1.5503 /usr/bin/oprofiled
1540 1.1130 /xfs
521 0.3765 pg-8.2.4/lib/plpgsql.so
441 0.3187 /cciss
374 0.2703 /oprofile

...

CPU: Core 2, speed 2666.76 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
unit mask of 0x00 (Unhalted core cycles) count 100000
samples % app name symbol name
85355 61.6887 pg-8.2.4/bin/postgres s_lock
9803 7.0849 pg-8.2.4/bin/postgres LWLockRelease
5535 4.0003 pg-8.2.4/bin/postgres LWLockAcquire
3792 2.7406 /lib/libc-2.3.6.so (no symbols)
3724 2.6915 pg-8.2.4/bin/postgres DropRelFileNodeBuffers
2145 1.5503 /usr/bin/oprofiled (no symbols)
2069 1.4953 pg-8.2.4/bin/postgres GetSnapshotData
1540 1.1130 /xfs (no symbols)
1246 0.9005 pg-8.2.4/bin/postgres hash_search_with_hash_value
1052 0.7603 pg-8.2.4/bin/postgres AllocSetAlloc
1015 0.7336 pg-8.2.4/bin/postgres heapgettup
879 0.6353 pg-8.2.4/bin/postgres hash_any
862 0.6230 /usr/src/linux-2.6.22.15/vmlinux mwait_idle
740 0.5348 pg-8.2.4/bin/postgres hash_seq_search
674 0.4871 pg-8.2.4/bin/postgres HeapTupleSatisfiesNow
557 0.4026 pg-8.2.4/bin/postgres SIGetDataEntry
552 0.3989 pg-8.2.4/bin/postgres equal
469 0.3390 pg-8.2.4/bin/postgres SearchCatCache
441 0.3187 /cciss (no symbols)
433 0.3129 /usr/src/linux-2.6.22.15/vmlinux find_busiest_group
413 0.2985 pg-8.2.4/bin/postgres PinBuffer
393 0.2840 pg-8.2.4/bin/postgres MemoryContextAllocZeroAligned
374 0.2703 /oprofile (no symbols)
275 0.1988 pg-8.2.4/bin/postgres ExecInitExpr
253 0.1829 pg-8.2.4/bin/postgres base_yyparse
206 0.1489 pg-8.2.4/bin/postgres CatalogCacheFlushRelation
201 0.1453 pg-8.2.4/bin/postgres MemoryContextAlloc
194 0.1402 pg-8.2.4/bin/postgres _bt_compare
188 0.1359 /nf_conntrack (no symbols)
158 0.1142 /bnx2 (no symbols)
147 0.1062 pg-8.2.4/bin/postgres pgstat_initstats
139 0.1005 pg-8.2.4/bin/postgres fmgr_info_cxt_security
132 0.0954 /usr/src/linux-2.6.22.15/vmlinux task_rq_lock
131 0.0947 /bin/bash (no symbols)
129 0.0932 pg-8.2.4/bin/postgres AllocSetFree
125 0.0903 pg-8.2.4/bin/postgres ReadBuffer
124 0.0896 pg-8.2.4/bin/postgres MemoryContextCreate
124 0.0896 pg-8.2.4/bin/postgres SyncOneBuffer
124 0.0896 pg-8.2.4/bin/postgres XLogInsert
123 0.0889 pg-8.2.4/bin/postgres _equalAggref
122 0.0882 pg-8.2.4/bin/postgres HeapTupleSatisfiesSnapshot
112 0.0809 pg-8.2.4/bin/postgres copyObject
102 0.0737 pg-8.2.4/bin/postgres UnpinBuffer
99 0.0716 pg-8.2.4/bin/postgres _SPI_execute_plan
99 0.0716 pg-8.2.4/bin/postgres nocachegetattr
98 0.0708 /usr/src/linux-2.6.22.15/vmlinux __wake_up_bit
97 0.0701 pg-8.2.4/bin/postgres TransactionIdIsInProgress
94 0.0679 pg-8.2.4/bin/postgres check_stack_depth
93 0.0672 pg-8.2.4/bin/postgres base_yylex
91 0.0658 pg-8.2.4/bin/postgres pfree
89 0.0643 pg-8.2.4/lib/plpgsql.so exec_stmts
86 0.0622 /usr/src/linux-2.6.22.15/vmlinux __switch_to
85 0.0614 pg-8.2.4/bin/postgres LockAcquire
83 0.0600 pg-8.2.4/bin/postgres FunctionCall2
82 0.0593 pg-8.2.4/bin/postgres ExecInitAgg
82 0.0593 /usr/src/linux-2.6.22.15/vmlinux system_call
79 0.0571 pg-8.2.4/bin/postgres pgstat_write_statsfile
77 0.0557 pg-8.2.4/bin/postgres heap_getsysattr
73 0.0528 pg-8.2.4/bin/postgres .plt
72 0.0520 /lib/ld-2.3.6.so (no symbols)
71 0.0513 pg-8.2.4/bin/postgres SearchSysCache
71 0.0513 pg-8.2.4/bin/postgres _bt_checkkeys
68 0.0491 /usr/src/linux-2.6.22.15/vmlinux apic_timer_interrupt
67 0.0484 pg-8.2.4/bin/postgres slot_deform_tuple
64 0.0463 pg-8.2.4/bin/postgres TupleDescInitEntry
64 0.0463 pg-8.2.4/bin/postgres newarc
63 0.0455 /usr/src/linux-2.6.22.15/vmlinux __wake_up
62 0.0448 pg-8.2.4/bin/postgres LocalExecuteInvalidationMessage
61 0.0441 /usr/src/linux-2.6.22.15/vmlinux try_to_wake_up
60 0.0434 pg-8.2.4/bin/postgres ReceiveSharedInvalidMessages
59 0.0426 pg-8.2.4/bin/postgres ExecutorStart
58 0.0419 pg-8.2.4/bin/postgres DirectFunctionCall1
58 0.0419 pg-8.2.4/bin/postgres ScanKeywordLookup
57 0.0412 pg-8.2.4/bin/postgres hash_search
56 0.0405 pg-8.2.4/bin/postgres CatalogCacheComputeHashValue
56 0.0405 pg-8.2.4/bin/postgres ExecProject
55 0.0398 pg-8.2.4/bin/postgres _bt_first
54 0.0390 pg-8.2.4/lib/plpgsql.so exec_eval_simple_expr
53 0.0383 pg-8.2.4/bin/postgres AllocSetDelete
53 0.0383 pg-8.2.4/bin/postgres CleanupTempFiles
52 0.0376 pg-8.2.4/bin/postgres ExecCreateTupleTable
52 0.0376 pg-8.2.4/lib/plpgsql.so copy_plpgsql_datum
49 0.0354 pg-8.2.4/bin/postgres MemoryContextAllocZero
49 0.0354 pg-8.2.4/bin/postgres SIDelExpiredDataEntries
49 0.0354 pg-8.2.4/bin/postgres fix_opfuncids_walker
47 0.0340 pg-8.2.4/bin/postgres expression_tree_walker
46 0.0332 pg-8.2.4/bin/postgres LockBuffer
45 0.0325 pg-8.2.4/bin/postgres lappend
45 0.0325 /usr/src/linux-2.6.22.15/vmlinux do_IRQ
44 0.0318 pg-8.2.4/bin/postgres LockReleaseAll
43 0.0311 pg-8.2.4/bin/postgres ExecutorRun
43 0.0311 pg-8.2.4/bin/postgres exprTypmod
42 0.0304 pg-8.2.4/bin/postgres ExecClearTuple
42 0.0304 pg-8.2.4/bin/postgres heap_fill_tuple
41 0.0296 pg-8.2.4/bin/postgres ExecIndexBuildScanKeys
40 0.0289 pg-8.2.4/bin/postgres _bt_readpage
40 0.0289 pg-8.2.4/lib/plpgsql.so plpgsql_exec_function
40 0.0289 /usr/src/linux-2.6.22.15/vmlinux lock_timer_base
39 0.0282 pg-8.2.4/bin/postgres heap_release_fetch
38 0.0275 pg-8.2.4/bin/postgres ExecEvalVar
38 0.0275 pg-8.2.4/bin/postgres ExecTypeFromTLInternal
37 0.0267 pg-8.2.4/bin/postgres LockReassignCurrentOwner
37 0.0267 /usr/src/linux-2.6.22.15/vmlinux scheduler_tick
36 0.0260 pg-8.2.4/bin/postgres ReleaseAndReadBuffer
36 0.0260 pg-8.2.4/bin/postgres _bt_checkpage
36 0.0260 pg-8.2.4/bin/postgres freearc
36 0.0260 pg-8.2.4/bin/postgres heapgetpage
36 0.0260 /usr/src/linux-2.6.22.15/vmlinux resched_task
35 0.0253 pg-8.2.4/bin/postgres PGSemaphoreLock
35 0.0253 pg-8.2.4/bin/postgres optimize
35 0.0253 pg-8.2.4/bin/postgres slot_getattr
35 0.0253 /usr/src/linux-2.6.22.15/vmlinux effective_prio
35 0.0253 /usr/src/linux-2.6.22.15/vmlinux hrtimer_run_queues
34 0.0246 /ip_tables (no symbols)
34 0.0246 pg-8.2.4/bin/postgres ExecCountSlotsNode
34 0.0246 pg-8.2.4/bin/postgres ExecMakeFunctionResultNoSets
34 0.0246 pg-8.2.4/bin/postgres ResourceOwnerForgetBuffer
34 0.0246 pg-8.2.4/bin/postgres TransactionIdPrecedes
34 0.0246 pg-8.2.4/bin/postgres _bt_getroot
34 0.0246 pg-8.2.4/lib/plpgsql.so exec_stmt_block
34 0.0246 /usr/src/linux-2.6.22.15/vmlinux do_page_fault
34 0.0246 /usr/src/linux-2.6.22.15/vmlinux run_rebalance_domains
33 0.0239 pg-8.2.4/bin/postgres AtEOXact_GUC
33 0.0239 /usr/src/linux-2.6.22.15/vmlinux __exit_idle
33 0.0239 /usr/src/linux-2.6.22.15/vmlinux sched_clock
32 0.0231 pg-8.2.4/bin/postgres _bt_moveright
32 0.0231 pg-8.2.4/bin/postgres compact
32 0.0231 /usr/src/linux-2.6.22.15/vmlinux __mod_timer
31 0.0224 pg-8.2.4/bin/postgres CreateExprContext
31 0.0224 pg-8.2.4/bin/postgres ExecInitNode
31 0.0224 pg-8.2.4/bin/postgres ExecMakeFunctionResult
31 0.0224 pg-8.2.4/bin/postgres ReleaseCatCache
31 0.0224 pg-8.2.4/bin/postgres exprType
30 0.0217 pg-8.2.4/bin/postgres _bt_binsrch
30 0.0217 pg-8.2.4/bin/postgres new_tail_cell
30 0.0217 /usr/src/linux-2.6.22.15/vmlinux run_timer_softirq
29 0.0210 pg-8.2.4/bin/postgres ExecAssignScanProjectionInfo
29 0.0210 pg-8.2.4/bin/postgres heap_form_tuple
28 0.0202 /usr/lib/gconv/ISO8859-1.so (no symbols)
28 0.0202 pg-8.2.4/bin/postgres AllocSetContextCreate
28 0.0202 pg-8.2.4/bin/postgres ResourceOwnerForgetCatCacheRef
28 0.0202 pg-8.2.4/bin/postgres hashoid
28 0.0202 pg-8.2.4/bin/postgres new_list
27 0.0195 pg-8.2.4/bin/postgres CreateTemplateTupleDesc
27 0.0195 pg-8.2.4/bin/postgres MemoryContextStrdup
27 0.0195 pg-8.2.4/bin/postgres SPI_execute_plan
27 0.0195 pg-8.2.4/bin/postgres btint4cmp
27 0.0195 /usr/src/linux-2.6.22.15/vmlinux IRQ0xa9_interrupt
26 0.0188 pg-8.2.4/bin/postgres ExecBuildProjectionInfo
26 0.0188 pg-8.2.4/bin/postgres ExecDropTupleTable
26 0.0188 pg-8.2.4/bin/postgres ExecEndNode
26 0.0188 pg-8.2.4/bin/postgres ExecEvalParam
26 0.0188 pg-8.2.4/bin/postgres FreeExecutorState
26 0.0188 pg-8.2.4/bin/postgres MemoryContextReset
26 0.0188 /usr/src/linux-2.6.22.15/vmlinux smp_apic_timer_interrupt
25 0.0181 pg-8.2.4/bin/postgres ExecInitSubPlan
25 0.0181 pg-8.2.4/bin/postgres FuncnameGetCandidates
25 0.0181 pg-8.2.4/bin/postgres MemoryContextDelete
25 0.0181 pg-8.2.4/bin/postgres SearchCatCacheList
25 0.0181 /usr/src/linux-2.6.22.15/vmlinux handle_edge_irq
25 0.0181 /usr/src/linux-2.6.22.15/vmlinux msecs_to_jiffies
24 0.0173 /nf_conntrack_ipv4 (no symbols)
24 0.0173 pg-8.2.4/bin/postgres ExecInitIndexScan
24 0.0173 pg-8.2.4/bin/postgres ExecProcNode
24 0.0173 pg-8.2.4/bin/postgres InternalCreateExecutorState
24 0.0173 pg-8.2.4/bin/postgres PostgresMain
24 0.0173 pg-8.2.4/bin/postgres RelationIncrementReferenceCount
24 0.0173 pg-8.2.4/bin/postgres heapgettup_pagemode
24 0.0173 /usr/src/linux-2.6.22.15/vmlinux __wake_up_common
24 0.0173 /usr/src/linux-2.6.22.15/vmlinux get_task_mm
24 0.0173 /usr/src/linux-2.6.22.15/vmlinux page_waitqueue
23 0.0166 pg-8.2.4/bin/postgres ExecCheckRTPerms
23 0.0166 pg-8.2.4/bin/postgres TransactionIdFollowsOrEquals
23 0.0166 pg-8.2.4/bin/postgres internal_putbytes
23 0.0166 pg-8.2.4/bin/postgres scanner_init
23 0.0166 pg-8.2.4/lib/plpgsql.so exec_stmt_execsql
23 0.0166 /usr/src/linux-2.6.22.15/vmlinux IRQ0xc1_interrupt
23 0.0166 /usr/src/linux-2.6.22.15/vmlinux __do_softirq
22 0.0159 pg-8.2.4/bin/postgres AfterTriggerBeginXact
22 0.0159 pg-8.2.4/bin/postgres GetTransactionSnapshot
22 0.0159 pg-8.2.4/bin/postgres ResourceOwnerReleaseInternal
22 0.0159 pg-8.2.4/bin/postgres lcons
22 0.0159 pg-8.2.4/bin/postgres markcanreach
22 0.0159 pg-8.2.4/lib/plpgsql.so exec_eval_datum
22 0.0159 pg-8.2.4/lib/plpgsql.so exec_eval_expr
22 0.0159 pg-8.2.4/lib/utf8_and_win.so win_to_utf8
22 0.0159 /usr/src/linux-2.6.22.15/vmlinux enqueue_task
22 0.0159 /usr/src/linux-2.6.22.15/vmlinux math_state_restore
21 0.0152 pg-8.2.4/bin/postgres AtCommit_Notify
21 0.0152 pg-8.2.4/bin/postgres CreateTupleDescCopy
21 0.0152 pg-8.2.4/bin/postgres ExecScan
21 0.0152 pg-8.2.4/bin/postgres ResourceOwnerEnlargeCatCacheRefs
21 0.0152 pg-8.2.4/bin/postgres ResourceOwnerForgetRelationRef
21 0.0152 pg-8.2.4/bin/postgres ResourceOwnerRememberCatCacheRef
21 0.0152 pg-8.2.4/bin/postgres SHMQueueInsertBefore
21 0.0152 pg-8.2.4/bin/postgres exec_simple_query
21 0.0152 pg-8.2.4/bin/postgres get_hash_value
21 0.0152 pg-8.2.4/bin/postgres index_getnext
21 0.0152 pg-8.2.4/bin/postgres next_token
21 0.0152 pg-8.2.4/bin/postgres pgstat_report_activity
21 0.0152 pg-8.2.4/lib/plpgsql.so exec_run_select
20 0.0145 pg-8.2.4/bin/postgres AllocSetReset
20 0.0145 pg-8.2.4/bin/postgres AtProcExit_Buffers
20 0.0145 pg-8.2.4/bin/postgres LocalToUtf
20 0.0145 pg-8.2.4/bin/postgres index_getprocinfo
20 0.0145 pg-8.2.4/bin/postgres spi_dest_startup
20 0.0145 pg-8.2.4/bin/postgres tag_hash
20 0.0145 /usr/src/linux-2.6.22.15/vmlinux sysret_check
19 0.0137 pg-8.2.4/bin/postgres Async_Unlisten
19 0.0137 pg-8.2.4/bin/postgres FileSeek
19 0.0137 pg-8.2.4/bin/postgres PortalStart
19 0.0137 pg-8.2.4/bin/postgres heap_open
19 0.0137 pg-8.2.4/bin/postgres pg_mblen
19 0.0137 /usr/src/linux-2.6.22.15/vmlinux prepare_to_wait
18 0.0130 /iptable_filter (no symbols)
18 0.0130 /iptable_nat (no symbols)
18 0.0130 /lib/libm-2.3.6.so (no symbols)
18 0.0130 pg-8.2.4/bin/postgres CopySnapshot
18 0.0130 pg-8.2.4/bin/postgres ExecSetSlotDescriptor
18 0.0130 pg-8.2.4/bin/postgres SPI_connect
18 0.0130 pg-8.2.4/bin/postgres cleartraverse
18 0.0130 pg-8.2.4/bin/postgres pg_mbcliplen
18 0.0130 pg-8.2.4/bin/postgres strlcpy
18 0.0130 pg-8.2.4/lib/plpgsql.so plpgsql_compile
18 0.0130 /usr/src/linux-2.6.22.15/vmlinux account_system_time
18 0.0130 /usr/src/linux-2.6.22.15/vmlinux bit_waitqueue
17 0.0123 pg-8.2.4/bin/postgres AtEOXact_MultiXact
17 0.0123 pg-8.2.4/bin/postgres CreateTupleDescCopyConstr
17 0.0123 pg-8.2.4/bin/postgres ExecAgg
17 0.0123 pg-8.2.4/bin/postgres ExecResult
17 0.0123 pg-8.2.4/bin/postgres FlushBuffer
17 0.0123 pg-8.2.4/bin/postgres RelationGetIndexScan
17 0.0123 pg-8.2.4/bin/postgres SHMQueueDelete
17 0.0123 pg-8.2.4/bin/postgres SendRowDescriptionMessage
17 0.0123 pg-8.2.4/bin/postgres _bt_search
17 0.0123 pg-8.2.4/bin/postgres _copyTargetEntry
17 0.0123 pg-8.2.4/bin/postgres heap_getnext
17 0.0123 pg-8.2.4/bin/postgres markreachable
17 0.0123 pg-8.2.4/bin/postgres miss
17 0.0123 pg-8.2.4/bin/postgres scanRTEForColumn
17 0.0123 pg-8.2.4/lib/plpgsql.so exec_assign_value
17 0.0123 /usr/src/linux-2.6.22.15/vmlinux do_softirq
16 0.0116 pg-8.2.4/bin/postgres ExecEvalConst
16 0.0116 pg-8.2.4/bin/postgres ExecReScan
16 0.0116 pg-8.2.4/bin/postgres ExecSetParamPlan
16 0.0116 pg-8.2.4/bin/postgres downcase_truncate_identifier
16 0.0116 pg-8.2.4/bin/postgres transformExpr
16 0.0116 pg-8.2.4/bin/postgres varstr_cmp
16 0.0116 pg-8.2.4/lib/plpgsql.so plpgsql_call_handler
16 0.0116 pg-8.2.4/lib/plpgsql.so plpgsql_estate_setup
16 0.0116 /usr/src/linux-2.6.22.15/vmlinux cpu_idle
15 0.0108 pg-8.2.4/bin/postgres CommitTransactionCommand
15 0.0108 pg-8.2.4/bin/postgres ExecEvalFuncArgs
15 0.0108 pg-8.2.4/bin/postgres GrantLockLocal
15 0.0108 pg-8.2.4/bin/postgres PrepareToInvalidateCacheTuple
15 0.0108 pg-8.2.4/bin/postgres UtfToLocal
15 0.0108 pg-8.2.4/bin/postgres cost_nestloop
15 0.0108 pg-8.2.4/bin/postgres errstart
15 0.0108 pg-8.2.4/bin/postgres index_open
15 0.0108 pg-8.2.4/bin/postgres init_fcache
15 0.0108 pg-8.2.4/bin/postgres list_delete_cell
14 0.0101 pg-8.2.4/bin/postgres ExecStoreTuple
14 0.0101 pg-8.2.4/bin/postgres ReadNewTransactionId
14 0.0101 pg-8.2.4/bin/postgres ReleaseBuffer
14 0.0101 pg-8.2.4/bin/postgres ResourceOwnerRememberBuffer
14 0.0101 pg-8.2.4/bin/postgres SHMQueueNext
14 0.0101 pg-8.2.4/bin/postgres btgettuple
14 0.0101 pg-8.2.4/bin/postgres duptraverse
14 0.0101 pg-8.2.4/bin/postgres heap_compute_data_size
14 0.0101 pg-8.2.4/bin/postgres index_rescan
14 0.0101 pg-8.2.4/bin/postgres list_copy
14 0.0101 pg-8.2.4/bin/postgres smgrclosenode
14 0.0101 pg-8.2.4/lib/plpgsql.so .plt
14 0.0101 /usr/src/linux-2.6.22.15/vmlinux idle_cpu
14 0.0101 /usr/src/linux-2.6.22.15/vmlinux run_workqueue
13 0.0094 pg-8.2.4/bin/postgres Async_Listen
13 0.0094 pg-8.2.4/bin/postgres DLMoveToFront
13 0.0094 pg-8.2.4/bin/postgres FreeExprContext
13 0.0094 pg-8.2.4/bin/postgres IndexNext
13 0.0094 pg-8.2.4/bin/postgres LockRelease
13 0.0094 pg-8.2.4/bin/postgres ResourceOwnerEnlargeBuffers
13 0.0094 pg-8.2.4/bin/postgres ShutdownExprContext
13 0.0094 pg-8.2.4/bin/postgres cleanup
13 0.0094 pg-8.2.4/bin/postgres datumCopy
13 0.0094 pg-8.2.4/bin/postgres grouping_planner
13 0.0094 pg-8.2.4/bin/postgres newstate
13 0.0094 pg-8.2.4/bin/postgres pgstat_start
13 0.0094 pg-8.2.4/bin/postgres spi_printtup
13 0.0094 pg-8.2.4/bin/postgres transformStmt
12 0.0087 pg-8.2.4/bin/postgres ExecEvalOper
12 0.0087 pg-8.2.4/bin/postgres ExecQual
12 0.0087 pg-8.2.4/bin/postgres IsSharedRelation
12 0.0087 pg-8.2.4/bin/postgres RelationIdGetRelation
12 0.0087 pg-8.2.4/bin/postgres StartTransactionCommand
12 0.0087 pg-8.2.4/bin/postgres _bt_insertonpg
12 0.0087 pg-8.2.4/bin/postgres advance_aggregates
12 0.0087 pg-8.2.4/bin/postgres getvacant
12 0.0087 pg-8.2.4/bin/postgres list_delete_ptr
12 0.0087 pg-8.2.4/bin/postgres namestrcpy
12 0.0087 pg-8.2.4/bin/postgres oideq
12 0.0087 /usr/src/linux-2.6.22.15/vmlinux local_bh_enable_ip
12 0.0087 /usr/src/linux-2.6.22.15/vmlinux sys_rt_sigreturn
11 0.0080 pg-8.2.4/bin/postgres CommitTransaction
11 0.0080 pg-8.2.4/bin/postgres ExecEvalNullTest
11 0.0080 pg-8.2.4/bin/postgres LockRelationOid
11 0.0080 pg-8.2.4/bin/postgres ProcessUtility
11 0.0080 pg-8.2.4/bin/postgres UnGrantLock
11 0.0080 pg-8.2.4/bin/postgres XidInSnapshot
11 0.0080 pg-8.2.4/bin/postgres _bt_doinsert
11 0.0080 pg-8.2.4/bin/postgres _bt_steppage
11 0.0080 pg-8.2.4/bin/postgres appendBinaryStringInfo
11 0.0080 pg-8.2.4/bin/postgres okcolors
11 0.0080 pg-8.2.4/bin/postgres pq_getmessage
11 0.0080 pg-8.2.4/lib/plpgsql.so exec_move_row
11 0.0080 /usr/src/linux-2.6.22.15/vmlinux clocksource_get_next
11 0.0080 /usr/src/linux-2.6.22.15/vmlinux handle_IRQ_event
11 0.0080 /usr/src/linux-2.6.22.15/vmlinux local_bh_enable
11 0.0080 /usr/src/linux-2.6.22.15/vmlinux mod_timer
10 0.0072 pg-8.2.4/bin/postgres CommandEndInvalidationMessages
10 0.0072 pg-8.2.4/bin/postgres CopyTriggerDesc
10 0.0072 pg-8.2.4/bin/postgres ExecAssignExprContext
10 0.0072 pg-8.2.4/bin/postgres ExecEvalScalarArrayOp
10 0.0072 pg-8.2.4/bin/postgres FunctionCall5
10 0.0072 pg-8.2.4/bin/postgres GetCurrentSubTransactionId
10 0.0072 pg-8.2.4/bin/postgres RecordTransactionCommit
10 0.0072 pg-8.2.4/bin/postgres StrategyGetBuffer
10 0.0072 pg-8.2.4/bin/postgres _bt_getbuf
10 0.0072 pg-8.2.4/bin/postgres _bt_preprocess_keys
10 0.0072 pg-8.2.4/bin/postgres eval_const_expressions_mutator
10 0.0072 pg-8.2.4/bin/postgres index_beginscan_internal
10 0.0072 pg-8.2.4/bin/postgres new_head_cell
10 0.0072 pg-8.2.4/bin/postgres perform_default_encoding_conversion
10 0.0072 pg-8.2.4/bin/postgres pg_regcomp
10 0.0072 pg-8.2.4/lib/plpgsql.so exec_eval_boolean
10 0.0072 pg-8.2.4/lib/plpgsql.so exec_stmt_if
10 0.0072 /usr/src/linux-2.6.22.15/vmlinux __rcu_pending
10 0.0072 /usr/src/linux-2.6.22.15/vmlinux __rcu_process_callbacks