Re: PostgreSQL 14.4 ERROR: out of memory issues

Hello,

čt 14. 7. 2022 v 23:31 odesílatel Aleš Zelený <zeleny(dot)ales(at)gmail(dot)com>
napsal:

>
>
> Dne čt 14. 7. 2022 23:11 uživatel Tomas Vondra <
> tomas(dot)vondra(at)enterprisedb(dot)com> napsal:
>
>>
>>
>> On 7/14/22 21:25, Aleš Zelený wrote:
>> >
>> > st 13. 7. 2022 v 2:20 odesílatel Michael Paquier <michael(at)paquier(dot)xyz
>> > <mailto:michael(at)paquier(dot)xyz>> napsal:
>> >
>> > On Mon, Jul 11, 2022 at 10:50:23AM +0200, Aleš Zelený wrote:
>> > > So far, it has happened three times (during a single week) from
>> > the 14.3 ->
>> > > 14.4 upgrade, before 14.4 we haven't suffered from such an issue.
>> > >
>> > > Questions:
>> > > 1) Can we safely downgrade from 14.4 to 14.3 by shutting down the
>> > instance
>> > > and reinstalling 14.3 PG packages (to prove, that the issue
>> > disappear)?
>> > > 2) What is the best way to diagnose what is the root cause?
>> >
>> > Hmm. 14.4 has nothing in its release notes that would point to a
>> > change in the vacuum or autovacuum's code paths:
>> > https://www.postgresql.org/docs/14/release-14-4.html#id-1.11.6.5.4
>> > <https://www.postgresql.org/docs/14/release-14-4.html#id-1.11.6.5.4
>> >
>> >
>> > There is nothing specific after a look at the changes as of, and I
>> am
>> > not grabbing anything that would imply a change in memory context
>> > handling either:
>> > `git log --stat REL_14_3..REL_14_4`
>> > `git diff REL_14_3..REL_14_4 -- *.c`
>> >
>> > Saying that, you should be able to downgrade safely as there are no
>> > changes in WAL format or such that would break things. Saying that,
>> > the corruption issue caused by CONCURRENTLY is something you'd still
>> > have to face.
>> >
>> >
>> > Thanks, good to know that, we can use it for a test case, since we
>> > already hit the CONCURRENTLY bug on 14.3.
>> >
>> > > 2022-07-02 14:48:07 CEST [3930]: [3-1] user=,db=,host=,app=
>> > ERROR: out of
>> > > memory
>> > > 2022-07-02 14:48:07 CEST [3930]: [4-1] user=,db=,host=,app=
>> > DETAIL: Failed
>> > > on request of size 152094068 in memory context
>> > "TopTransactionContext".
>> > > 2022-07-02 14:48:07 CEST [3930]: [5-1] user=,db=,host=,app=
>> CONTEXT:
>> > > automatic vacuum of table "prematch.replication.tab_queue_tmp"
>> >
>> > This is the interesting part. Do you happen to use logical
>> > replication in a custom C++ plugin?
>> >
>> >
>> > We are using logical replication to other instances (pg_output) and
>> > decoderbufs
>> > https://github.com/debezium/postgres-decoderbufs
>> > <https://github.com/debezium/postgres-decoderbufs> for other
>> applications.
>> >
>>
>> This is probably just a red herring - std:bad_alloc is what the process
>> that runs into the overcommit limit gets. But the real issue (e.g.
>> memory leak) is likely somewhere else - different part of the code,
>> different process ...
>>
>> > ...
>> >
>> > Checking the RssAnon from proc/pid/status I've found some points where
>> > RssAnon memory usage grew very steep for a minute, but no "suspicious"
>> > queries/arguments were found in the instance logfile.
>> >
>> > Any hint, on how to get the root cause would be appreciated since so far
>> > I've failed to isolate the issue reproducible testcase.
>> > At least I hope that looking for the RssAnon process memory is an
>> > appropriate metric, if not, let me know and I'll try to update the
>> > monitoring to get the root cause.
>> >
>> > I can imagine a workaround with client application regular reconnect...,
>> > but u to 14.3 it works, so I'd like to fix the issue either on our
>> > application side or at PG side if it is a PG problem.
>> >
>>
>> I think it's be interesting to get memory context stats from the
>> processes consuming a lot of memory. If you know which processes are
>> suspect (and it seems you know, bacause if a reconnect helps it's the
>> backend handling the connection), you can attach a debugger and do
>>
>> $ gdb -p $PID
>> call MemoryContextStats(TopMemoryContext)
>>
>> which will log info about memory contexts, just like autovacuum.
>> Hopefully that tells us memory context is bloated, and that might point
>> us to particular part of the code.
>>
>
after some time, I've found a process consuming over 1GB of memory"
-bash-4.2$ grep RssAnon /proc/*/status | sort -nk2 | tail
/proc/12156/status:RssAnon: 902984 kB
/proc/8303/status:RssAnon: 912848 kB
/proc/31187/status:RssAnon: 920424 kB
/proc/16469/status:RssAnon: 922864 kB
/proc/13769/status:RssAnon: 925824 kB
/proc/15449/status:RssAnon: 932452 kB
/proc/26144/status:RssAnon: 936892 kB
/proc/22570/status:RssAnon: 937920 kB
/proc/2910/status:RssAnon: 1039124 kB
/proc/17048/status:RssAnon: 1053952 kB

Here are memory contexts for PID 17048:

TopMemoryContext: 422592 total in 14 blocks; 42536 free (169 chunks);
380056 used
pgstat TabStatusArray lookup hash table: 16384 total in 2 blocks; 7520
free (3 chunks); 8864 used
PL/pgSQL function: 16384 total in 2 blocks; 5832 free (1 chunks); 10552
used: feeds_api.update_match(bigint,date,time without time zone)
PL/pgSQL function: 16384 total in 2 blocks; 2080 free (1 chunks); 14304
used: opportunities.tr_taboddstopchangeshistory_oddstopchangestime()
PL/pgSQL function: 16384 total in 2 blocks; 8576 free (1 chunks); 7808
used: opportunities.tr_tabodds_risklimits()
PL/pgSQL function: 16384 total in 2 blocks; 2072 free (1 chunks); 14312
used: opportunities.tr_taboddshistory_oddshistorytime()
PL/pgSQL function: 32768 total in 3 blocks; 6488 free (1 chunks); 26280
used: opportunities.fix_last_minute_odds(bigint,bigint)
PL/pgSQL function: 65536 total in 4 blocks; 26624 free (1 chunks); 38912
used:
opportunities.create_odds_tip_longname_lang(bigint,integer,integer,integer)
PL/pgSQL function: 16384 total in 2 blocks; 3472 free (1 chunks); 12912
used:
feeds_api.create_odds(bigint,integer,numeric,numeric,numeric,numeric,numeric,integer,jsonb)
PL/pgSQL function: 65536 total in 4 blocks; 29648 free (2 chunks); 35888
used:
feeds_api.create_opportunity(integer,bigint,bigint,integer,integer,integer,integer,integer,integer,i...
PL/pgSQL function: 16384 total in 2 blocks; 7576 free (1 chunks); 8808
used: feeds.tr_tabopptysparentdata_ismain()
PL/pgSQL function: 32768 total in 3 blocks; 11720 free (1 chunks); 21048
used: feeds_api.set_game_data(integer,bigint,bigint,character varying)
PL/pgSQL function: 32768 total in 3 blocks; 7712 free (1 chunks); 25056
used: opportunities_api.set_oppty_info(bigint,boolean)
PL/pgSQL function: 8192 total in 1 blocks; 3488 free (1 chunks); 4704
used: feeds_api.set_oppty_info_number(bigint)
PL/pgSQL function: 16384 total in 2 blocks; 6616 free (1 chunks); 9768
used: b5_offer.tr_tabexternalopportunities_checknotnull()
PL/pgSQL function: 32768 total in 3 blocks; 17928 free (5 chunks); 14840
used: opportunities.generate_league2oppty_correlations(bigint)
PL/pgSQL function: 131072 total in 5 blocks; 61648 free (6 chunks); 69424
used: opportunities.generate_league2league_correlations(bigint)
PL/pgSQL function: 16384 total in 2 blocks; 5616 free (1 chunks); 10768
used: opportunities.tr_tabopptytree_addcorrelations()
PL/pgSQL function: 65536 total in 4 blocks; 19864 free (1 chunks); 45672
used: opportunities.replication_151_tab_oppty_tree()
PL/pgSQL function: 16384 total in 2 blocks; 7568 free (1 chunks); 8816
used: greep.tr_shared_changedat()
PL/pgSQL function: 32768 total in 3 blocks; 2112 free (1 chunks); 30656
used:
feeds_api.create_match(integer,integer,integer,jsonb,bigint,integer,integer,integer,integer,jsonb,in...
HandleParallelMessages: 8192 total in 1 blocks; 7928 free (0 chunks); 264
used
pgqs_localhash: 21448 total in 2 blocks; 2616 free (0 chunks); 18832 used
PL/pgSQL function: 16384 total in 2 blocks; 2072 free (1 chunks); 14312
used: opportunities.tr_tabopptyhistory_opptyhistorytime()
PL/pgSQL function: 16384 total in 2 blocks; 2360 free (1 chunks); 14024
used: opportunities.tr_tabopportunitiesevaluationqueue_validtime()
PL/pgSQL function: 16384 total in 2 blocks; 2512 free (1 chunks); 13872
used: opportunities.tr_tabopportunitiesevaluationqueue_evaluatedtime()
PL/pgSQL function: 16384 total in 2 blocks; 2360 free (1 chunks); 14024
used: opportunities.tr_tabopportunitiesevaluationqueue_addedtime()
PL/pgSQL function: 16384 total in 2 blocks; 7512 free (1 chunks); 8872
used:
feeds_api.evaluate_opportunities(bigint[],integer,integer,integer,integer,integer)
Btree proof lookup cache: 8192 total in 1 blocks; 552 free (0 chunks);
7640 used
PL/pgSQL function: 8192 total in 1 blocks; 1384 free (1 chunks); 6808
used: feeds_api.change_odds_is_winning(bigint[],integer)
PL/pgSQL function: 32768 total in 3 blocks; 3160 free (1 chunks); 29608
used: opportunities.tr_tabopportunity_resultchange()
PL/pgSQL function: 16384 total in 2 blocks; 4424 free (1 chunks); 11960
used: opportunities.tr_tabopportunity_opptynamechange()
PL/pgSQL function: 16384 total in 2 blocks; 7448 free (1 chunks); 8936
used: opportunities.tr_tabopportunity_marketingstatechange()
PL/pgSQL function: 65536 total in 4 blocks; 27224 free (2 chunks); 38312
used: opportunities.tr_tabopportunity_history()
PL/pgSQL function: 32768 total in 3 blocks; 16224 free (3 chunks); 16544
used: opportunities.tr_tabopportunity_datetimechange()
PL/pgSQL function: 32768 total in 3 blocks; 15440 free (1 chunks); 17328
used: opportunities.tr_tabopportunity_addcorrelation()
PL/pgSQL function: 32768 total in 3 blocks; 15672 free (1 chunks); 17096
used: opportunities.tr_tabopportunities_notifycanchanges()
Sequence values: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
PL/pgSQL function: 262144 total in 6 blocks; 80744 free (0 chunks);
181400 used: opportunities.replication_151_tab_opportunities()
PL/pgSQL function: 16384 total in 2 blocks; 8904 free (5 chunks); 7480
used: betsys.lastuser_change()
PL/pgSQL function: 16384 total in 2 blocks; 2448 free (1 chunks); 13936
used: opportunities.tr_tabopportunities_opptyevaltime()
PL/pgSQL function: 65536 total in 4 blocks; 29064 free (1 chunks); 36472
used: opportunities.tr_tabopportunities_checkstatechange()
PL/pgSQL function: 16384 total in 2 blocks; 7568 free (1 chunks); 8816
used: greep.tr_shared_changedat()
PL/pgSQL function: 32768 total in 3 blocks; 15512 free (1 chunks); 17256
used: opportunities.tr_tabopportunities_opptyenddatetime()
PL/pgSQL function: 32768 total in 3 blocks; 16008 free (4 chunks); 16760
used: opportunities.tr_tabopportunity_checkstatechange_au()
PL/pgSQL function: 8192 total in 1 blocks; 1752 free (1 chunks); 6440
used: feeds_api.change_opportunities_state(bigint[],integer)
PL/pgSQL function: 65536 total in 4 blocks; 27640 free (1 chunks); 37896
used: opportunities.tr_tabodds_aggregs()
PL/pgSQL function: 16384 total in 2 blocks; 7128 free (1 chunks); 9256
used: opportunities.tr_tabodds_notifyoddsblocked()
PL/pgSQL function: 65536 total in 4 blocks; 31200 free (1 chunks); 34336
used: opportunities.tr_tabodds_history()
Record information cache: 8192 total in 1 blocks; 1576 free (0 chunks);
6616 used
PL/pgSQL function: 131072 total in 5 blocks; 65672 free (2 chunks); 65400
used: opportunities.replication_151_tab_odds()
RI compare cache: 16384 total in 2 blocks; 6656 free (3 chunks); 9728 used
RI query cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
RI constraint cache: 40648 total in 2 blocks; 2616 free (0 chunks); 38032
used
PL/pgSQL function: 16384 total in 2 blocks; 7920 free (1 chunks); 8464
used: opportunities.tr_tabodds_fixoddsvalue()
PLpgSQL cast info: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
PLpgSQL cast cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616
used
PL/pgSQL function: 16384 total in 2 blocks; 7568 free (5 chunks); 8816
used: greep.tr_shared_changedat()
CFuncHash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
TableSpace cache: 8192 total in 1 blocks; 2088 free (0 chunks); 6104 used
Operator lookup cache: 40960 total in 3 blocks; 18888 free (10 chunks);
22072 used
Type information cache: 24376 total in 2 blocks; 2616 free (0 chunks);
21760 used
Function stat entries: 16384 total in 2 blocks; 4576 free (2 chunks);
11808 used
Prepared Queries: 16384 total in 2 blocks; 6656 free (3 chunks); 9728 used
RowDescriptionContext: 8192 total in 1 blocks; 6888 free (0 chunks); 1304
used
MessageContext: 8192 total in 1 blocks; 6888 free (1 chunks); 1304 used
Operator class cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640
used
smgr relation table: 65536 total in 4 blocks; 20768 free (14 chunks);
44768 used
TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks);
264 used
Portal hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
TopPortalContext: 8192 total in 1 blocks; 7928 free (2 chunks); 264 used
Relcache by OID: 32768 total in 3 blocks; 13608 free (6 chunks); 19160
used
CacheMemoryContext: 2919800 total in 48 blocks; 218712 free (4 chunks);
2701088 used
CachedPlanSource: 4096 total in 3 blocks; 1496 free (0 chunks); 2600
used: SELECT * FROM feeds_api.change_odds_is_winning(_id_odds := $1,
_odds_is_winning := $2)
CachedPlanQuery: 4096 total in 3 blocks; 1568 free (3 chunks); 2528
used
CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used:
INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES
(currval('replication.tab_queue_d...
CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used:
INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES
(currval('replication.tab_queue_d...
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
tab_queue_tmp_pkey
CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used:
INSERT INTO replication.tab_queue_tmp (id_queue_data)
VALUES (currval('repli...
CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used:
INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES
(currval('replication.tab_queue_dat...
CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used:
INSERT INTO replication.tab_queue_tmp (id_queue_data)
VALUES (currval('repli...
CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used:
INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES
(currval('replication.tab_queue_dat...
CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used:
INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES
(currval('replication.tab_queue_dat...
index info: 4096 total in 3 blocks; 2264 free (2 chunks); 1832 used:
ix_tabopportunitiesevaluationqueue_idopportunity_pk_active_2
index info: 4096 total in 3 blocks; 2024 free (2 chunks); 2072 used:
ix_tabopportunitiesevaluationqueue_idopportunity_pk_active_1
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunitiesevaluationqueue_idodds
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
pk_tabopportunitiesevaluationqueue_idopportunitiesevaluationque
CachedPlan: 16384 total in 5 blocks; 3656 free (0 chunks); 12728 used:
INSERT INTO opportunities.tab_opportunities_evaluation_queue (id_odds,
evaluation_type, id_opportuni...
CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used:
INSERT INTO replication.tab_queue_data (queue_data, id_action_type,
id_table, id_replication_group) ...
CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used:
INSERT INTO replication.tab_queue_data (queue_data, id_action_type,
id_table, id_replication_group) ...
CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used:
INSERT INTO replication.tab_queue_data (queue_data, id_action_type,
id_table, id_replication_group) ...
CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used:
INSERT INTO replication.tab_queue_data (queue_data, id_action_type,
id_table, id_replication_group) ...
CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used:
INSERT INTO replication.tab_queue_data (queue_data, id_action_type,
id_table, id_replication_group) ...
CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used:
INSERT INTO replication.tab_queue_data (queue_data, id_action_type,
id_table, id_replication_group) ...
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
i_replication_tabqueuedata_idreplicationgroup
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
tab_queue_data_pkey
CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used:
INSERT INTO replication.tab_queue_data (queue_data, id_action_type,
id_table, id_replication_group) ...
CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used:
SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlan: 134336 total in 10 blocks; 42112 free (0 chunks); 92224
used: WITH opp AS MATERIALIZED ( SELECT DISTINCT
o2.id_opportunity FROM o...
CachedPlan: 65536 total in 7 blocks; 31968 free (0 chunks); 33568 used:
SELECT lc.contestant_name_lang
FROM opportunities.tab_op...
CachedPlan: 65536 total in 7 blocks; 31968 free (0 chunks); 33568 used:
SELECT lc.contestant_name_lang
FROM opportunities.tab_op...
CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used:
SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used:
SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used:
SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used:
SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used:
SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlan: 8192 total in 4 blocks; 3672 free (0 chunks); 4520 used:
EXISTS ( SELECT FROM
opportunities.tab_oppty_tree WHE...
CachedPlan: 65536 total in 7 blocks; 32224 free (0 chunks); 33312 used:
SELECT CASE WHEN SUBSTRING(op.marketing_states FROM 1 FOR 1) = '1'
AND op.oppty_end_date + op...
CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712
used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688
used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
CachedPlan: 4096 total in 3 blocks; 392 free (0 chunks); 3704 used:
INSERT INTO opportunities.tab_oppty_tree ( id_opportunity,
id_opportunity_parent...
CachedPlan: 35840 total in 6 blocks; 15344 free (1 chunks); 20496 used:
INSERT INTO opportunities.tab_opportunities ( id_event,
id_oppty_type, i...
CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712
used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688
used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
CachedPlan: 8192 total in 4 blocks; 3664 free (1 chunks); 4528 used:
NOT EXISTS ( SELECT FROM
opportunities.tab_oppty_tree WH...
CachedPlan: 16384 total in 5 blocks; 7960 free (2 chunks); 8424 used:
UPDATE opportunities.tab_opportunities SET oppty_info_number =
_info_number, id_op...
CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712
used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlanQuery: 8192 total in 4 blocks; 1504 free (3 chunks); 6688
used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
CachedPlan: 8192 total in 4 blocks; 616 free (2 chunks); 7576 used:
SELECT oppty_info_number, id_event, id_oppty_type
...
CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712
used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688
used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
CachedPlan: 134408 total in 10 blocks; 51528 free (0 chunks); 82880
used: INSERT INTO opportunities.tab_correlations (id_opportunity,
id_opportunity_2) SELECT crr...
CachedPlan: 277504 total in 9 blocks; 15680 free (0 chunks); 261824
used: INSERT INTO opportunities.tab_correlations (id_opportunity,
id_opportunity_2) -- id_league from ...
CachedPlan: 134144 total in 8 blocks; 30024 free (0 chunks); 104120
used: SELECT nm1.contestant_name,
COALESCE(nm2.contestant_name, nm1.contestant_name), ...
CachedPlan: 32768 total in 6 blocks; 2064 free (0 chunks); 30704 used:
SELECT evn.id_league FROM opportunities.tab_opportunities AS opp
JOIN opportunities.tab_e...
CachedPlan: 16384 total in 5 blocks; 6880 free (0 chunks); 9504 used:
NEW.id_opportunity <> NEW.id_opportunity_parent OR EXISTS (
SELECT ...
CachedPlan: 16384 total in 5 blocks; 4952 free (1 chunks); 11432 used:
(TG_OP='INSERT' AND CASE WHEN get_session_var('id_data_provider') = '12'
THEN FALSE ELSE (SELECT CAS...
CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712
used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688
used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712
used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688
used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
CachedPlan: 4096 total in 3 blocks; 392 free (0 chunks); 3704 used:
INSERT INTO opportunities.tab_oppty_tree ( id_opportunity,
id_opportunity_parent...
CachedPlan: 33792 total in 6 blocks; 14192 free (0 chunks); 19600 used:
INSERT INTO opportunities.tab_opportunities ( id_event,
id_oppty_type, o...
CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used:
SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712
used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE
"id_opportunity" OPERATOR(pg_catalog....
CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688
used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
CachedPlan: 32768 total in 6 blocks; 9352 free (0 chunks); 23416 used:
EXISTS ( SELECT FROM new_items AS ni JOIN
old_items AS oi ON oi.id_opportu...
CachedPlan: 8192 total in 4 blocks; 3120 free (1 chunks); 5072 used:
OLD.oppty_name IS DISTINCT FROM NEW.oppty_name AND EXISTS (
SELECT FROM oppo...
CachedPlan: 9216 total in 4 blocks; 1408 free (1 chunks); 7808 used:
NEW.oppty_end_date + NEW.oppty_end_time > NOW() AND
(COALESCE(OLD.oppty_end_date + OLD.oppty...
index info: 2048 total in 2 blocks; 392 free (0 chunks); 1656 used:
ix_tabopptytree_idopportunityparent_temporary
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pk_tabopptytree_idopportunity
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopptytree_idopportunityparent
index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used:
pg_statistic_relid_att_inh_index
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used:
ix_tabopportunities_opptybetradarid_iddataprovider
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_opptyevaltimetz
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_opptyendtimestamptz
index info: 2048 total in 2 blocks; 184 free (0 chunks); 1864 used:
ix_tabopportunities_opptyenddatetime
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_idconstestant4
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_idconstestant3
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_opptyexternalcompleteid
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_opptybetradarid
index info: 4096 total in 3 blocks; 2024 free (2 chunks); 2072 used:
ix_tabopportunities_opptyenddatetime_idopptystate
index info: 2048 total in 2 blocks; 392 free (0 chunks); 1656 used:
ix_tabopportunities_opptyenddatetime_obsolete
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_opptyenddate
index info: 2048 total in 2 blocks; 264 free (0 chunks); 1784 used:
ix_tabopportunities_lastminute
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
ix_tabopportunities_info
index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used:
ix_tabopportunities_idopptytype_vsportsid_idevent
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_idopportunitytype
index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used:
ix_tabopportunities_idopportunitystate_opptyenddate_idevent
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used:
ix_tabopportunities_idopportunitystate_opptyenddate
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_idopportunitystate
index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used:
ix_tabopportunities_idopportunity_opptyenddate_opptyinfonumber
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used:
ix_tabopportunities_idevent_opptyenddate
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_idconstestant2
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_tabopportunities_idconstestant1
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used:
ix_tabopportunities_enddateinfonumber
index info: 2048 total in 2 blocks; 360 free (0 chunks); 1688 used:
idx_tabopportunities_opptyspecialvalues_season_id
index info: 2048 total in 2 blocks; 360 free (0 chunks); 1688 used:
idx_tabopportunities_opptyspecialvalues_match_id
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
pk_tabopportunities_idopportunity
CachedPlan: 16384 total in 5 blocks; 6712 free (0 chunks); 9672 used:
UPDATE opportunities.lst_oppty_numbers SET id_opportunity =
_id_opportunity WHERE id_oppty_n...
CachedPlan: 8192 total in 4 blocks; 3704 free (1 chunks); 4488 used:
UPDATE opportunities.lst_oppty_numbers SET
id_opportunity = NULL ...
index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used:
pg_toast_2619_index
index info: 2048 total in 2 blocks; 584 free (0 chunks); 1464 used:
ix_lstopptynumbers_freeopptynumbers
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used:
ix_lstopptynumbers_idopportunity
1455 more child contexts containing 6947040 total in 4141 blocks;
2312976 free (668 chunks); 4634064 used
WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks);
43408 used
PrivateRefCount: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
MdSmgr: 16384 total in 2 blocks; 5624 free (20 chunks); 10760 used
LOCALLOCK hash: 32768 total in 3 blocks; 12712 free (8 chunks); 20056 used
plpgsql_check - profiler context: 8192 total in 1 blocks; 7928 free (0
chunks); 264 used
plpgsql_check function execution statistics: 8192 total in 1 blocks;
2488 free (0 chunks); 5704 used
plpgsql_check function profiler local chunks: 8192 total in 1 blocks;
2488 free (0 chunks); 5704 used
plpgsql_check function profiler local cache: 8192 total in 1 blocks;
2488 free (0 chunks); 5704 used
plpgsql_check function cache: 8192 total in 1 blocks; 2616 free (0
chunks); 5576 used
Rendezvous variable hash: 8192 total in 1 blocks; 552 free (0 chunks);
7640 used
PLpgSQL function hash: 40864 total in 3 blocks; 2616 free (0 chunks);
38248 used
Timezones: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
ErrorContext: 8192 total in 1 blocks; 7928 free (8 chunks); 264 used
Grand total: 14312808 bytes in 4752 blocks; 3920880 free (1043 chunks);
10391928 used

I hope, this will help.

Some additional information:

-bash-4.2$ cat /proc/17048/status
Name: postmaster
Umask: 0077
State: S (sleeping)
Tgid: 17048
Ngid: 0
Pid: 17048
PPid: 4476
TracerPid: 0
Uid: 26 26 26 26
Gid: 26 26 26 26
FDSize: 1024
Groups: 26
VmPeak: 12212748 kB
VmSize: 12212284 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 9421940 kB
VmRSS: 9421940 kB
RssAnon: 1054764 kB
RssFile: 15796 kB
RssShmem: 8351380 kB
VmData: 1055404 kB
VmStk: 132 kB
VmExe: 7816 kB
VmLib: 85312 kB
VmPTE: 23264 kB
VmSwap: 0 kB
Threads: 1
SigQ: 0/160197
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000400000
SigIgn: 0000000001701800
SigCgt: 00000001a0006287
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000001fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Speculation_Store_Bypass: thread vulnerable
Cpus_allowed: ffffff
Cpus_allowed_list: 0-23
Mems_allowed:
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 4748002
nonvoluntary_ctxt_switches: 12237

-bash-4.2$ pmap 17048
17048: postgres: db-prematch: app_feed prematch 172.25.0.231(47890) idle
0000000000400000 7816K r-x-- postgres
0000000000da1000 4K r---- postgres
0000000000da2000 68K rw--- postgres
0000000000db3000 200K rw--- [ anon ]
0000000002d15000 504K rw--- [ anon ]
0000000002d93000 838092K rw--- [ anon ]
00007fd999777000 180232K rw--- [ anon ]
00007fd9a8d75000 32772K rw--- [ anon ]
00007fd9ac5d2000 1024K rw-s- PostgreSQL.380355184
00007fd9ac6d2000 196K rw-s- PostgreSQL.38763112
00007fd9ac703000 148K r-x-- libtinfo.so.5.9
00007fd9ac728000 2048K ----- libtinfo.so.5.9
00007fd9ac928000 16K r---- libtinfo.so.5.9
00007fd9ac92c000 4K rw--- libtinfo.so.5.9
00007fd9ac92d000 212K r-x-- libedit.so.0.0.42
00007fd9ac962000 2044K ----- libedit.so.0.0.42
00007fd9acb61000 12K r---- libedit.so.0.0.42
00007fd9acb64000 4K rw--- libedit.so.0.0.42
00007fd9acb65000 20K rw--- [ anon ]
00007fd9acb6a000 28K r-x-- libffi.so.6.0.1
00007fd9acb71000 2044K ----- libffi.so.6.0.1
00007fd9acd70000 4K r---- libffi.so.6.0.1
00007fd9acd71000 4K rw--- libffi.so.6.0.1
00007fd9acd72000 45952K r-x-- libLLVM-5.0.so
00007fd9afa52000 2048K ----- libLLVM-5.0.so
00007fd9afc52000 3072K r---- libLLVM-5.0.so
00007fd9aff52000 444K rw--- libLLVM-5.0.so
00007fd9affc1000 372K rw--- [ anon ]
00007fd9b001e000 116K r-x-- llvmjit.so
00007fd9b003b000 2048K ----- llvmjit.so
00007fd9b023b000 4K r---- llvmjit.so
00007fd9b023c000 4K rw--- llvmjit.so
00007fd9b023d000 1028K rw--- [ anon ]
00007fd9b033e000 48K r-x-- libnss_files-2.17.so
00007fd9b034a000 2044K ----- libnss_files-2.17.so
00007fd9b0549000 4K r---- libnss_files-2.17.so
00007fd9b054a000 4K rw--- libnss_files-2.17.so
00007fd9b054b000 24K rw--- [ anon ]
00007fd9b0551000 10827040K rw-s- zero (deleted)
00007fdc45299000 32K r-x-- libprotobuf-c.so.1
00007fdc452a1000 2044K ----- libprotobuf-c.so.1
00007fdc454a0000 4K r---- libprotobuf-c.so.1
00007fdc454a1000 4K rw--- libprotobuf-c.so.1
00007fdc454a2000 24K r-x-- decoderbufs.so
00007fdc454a8000 2048K ----- decoderbufs.so
00007fdc456a8000 4K r---- decoderbufs.so
00007fdc456a9000 4K rw--- decoderbufs.so
00007fdc456aa000 184K r-x-- plpgsql.so
00007fdc456d8000 2044K ----- plpgsql.so
00007fdc458d7000 8K r---- plpgsql.so
00007fdc458d9000 4K rw--- plpgsql.so
00007fdc458da000 192K r-x-- plpgsql_check.so
00007fdc4590a000 2044K ----- plpgsql_check.so
00007fdc45b09000 4K r---- plpgsql_check.so
00007fdc45b0a000 4K rw--- plpgsql_check.so
00007fdc45b0b000 24K r-x-- pg_wait_sampling.so
00007fdc45b11000 2044K ----- pg_wait_sampling.so
00007fdc45d10000 4K r---- pg_wait_sampling.so
00007fdc45d11000 4K rw--- pg_wait_sampling.so
00007fdc45d12000 20K r-x-- pg_stat_kcache.so
00007fdc45d17000 2044K ----- pg_stat_kcache.so
00007fdc45f16000 4K r---- pg_stat_kcache.so
00007fdc45f17000 4K rw--- pg_stat_kcache.so
00007fdc45f18000 16K rw--- [ anon ]
00007fdc45f1c000 32K r-x-- pg_qualstats.so
00007fdc45f24000 2044K ----- pg_qualstats.so
00007fdc46123000 4K r---- pg_qualstats.so
00007fdc46124000 4K rw--- pg_qualstats.so
00007fdc46125000 32K r-x-- pg_stat_statements.so
00007fdc4612d000 2044K ----- pg_stat_statements.so
00007fdc4632c000 4K r---- pg_stat_statements.so
00007fdc4632d000 4K rw--- pg_stat_statements.so
00007fdc4632e000 103692K r---- locale-archive
00007fdc4c871000 8K r-x-- libfreebl3.so
00007fdc4c873000 2044K ----- libfreebl3.so
00007fdc4ca72000 4K r---- libfreebl3.so
00007fdc4ca73000 4K rw--- libfreebl3.so
00007fdc4ca74000 60K r-x-- libbz2.so.1.0.6
00007fdc4ca83000 2044K ----- libbz2.so.1.0.6
00007fdc4cc82000 4K r---- libbz2.so.1.0.6
00007fdc4cc83000 4K rw--- libbz2.so.1.0.6
00007fdc4cc84000 92K r-x-- libelf-0.176.so
00007fdc4cc9b000 2044K ----- libelf-0.176.so
00007fdc4ce9a000 4K r---- libelf-0.176.so
00007fdc4ce9b000 4K rw--- libelf-0.176.so
00007fdc4ce9c000 384K r-x-- libpcre.so.1.2.0
00007fdc4cefc000 2048K ----- libpcre.so.1.2.0
00007fdc4d0fc000 4K r---- libpcre.so.1.2.0
00007fdc4d0fd000 4K rw--- libpcre.so.1.2.0
00007fdc4d0fe000 16K r-x-- libattr.so.1.1.0
00007fdc4d102000 2044K ----- libattr.so.1.1.0
00007fdc4d301000 4K r---- libattr.so.1.1.0
00007fdc4d302000 4K rw--- libattr.so.1.1.0
00007fdc4d303000 32K r-x-- libcrypt-2.17.so
00007fdc4d30b000 2044K ----- libcrypt-2.17.so
00007fdc4d50a000 4K r---- libcrypt-2.17.so
00007fdc4d50b000 4K rw--- libcrypt-2.17.so
00007fdc4d50c000 184K rw--- [ anon ]
00007fdc4d53a000 16K r-x-- libcap-ng.so.0.0.0
00007fdc4d53e000 2048K ----- libcap-ng.so.0.0.0
00007fdc4d73e000 4K r---- libcap-ng.so.0.0.0
00007fdc4d73f000 4K rw--- libcap-ng.so.0.0.0
00007fdc4d740000 312K r-x-- libdw-0.176.so
00007fdc4d78e000 2048K ----- libdw-0.176.so
00007fdc4d98e000 8K r---- libdw-0.176.so
00007fdc4d990000 4K rw--- libdw-0.176.so
00007fdc4d991000 16K r-x-- libgpg-error.so.0.10.0
00007fdc4d995000 2044K ----- libgpg-error.so.0.10.0
00007fdc4db94000 4K r---- libgpg-error.so.0.10.0
00007fdc4db95000 4K rw--- libgpg-error.so.0.10.0
00007fdc4db96000 500K r-x-- libgcrypt.so.11.8.2
00007fdc4dc13000 2044K ----- libgcrypt.so.11.8.2
00007fdc4de12000 4K r---- libgcrypt.so.11.8.2
00007fdc4de13000 12K rw--- libgcrypt.so.11.8.2
00007fdc4de16000 4K rw--- [ anon ]
00007fdc4de17000 144K r-x-- libselinux.so.1
00007fdc4de3b000 2044K ----- libselinux.so.1
00007fdc4e03a000 4K r---- libselinux.so.1
00007fdc4e03b000 4K rw--- libselinux.so.1
00007fdc4e03c000 8K rw--- [ anon ]
00007fdc4e03e000 16K r-x-- libcap.so.2.22
00007fdc4e042000 2044K ----- libcap.so.2.22
00007fdc4e241000 4K r---- libcap.so.2.22
00007fdc4e242000 4K rw--- libcap.so.2.22
00007fdc4e243000 20296K r-x-- libicudata.so.50.2
00007fdc4f615000 2044K ----- libicudata.so.50.2
00007fdc4f814000 4K r---- libicudata.so.50.2
00007fdc4f815000 4K rw--- libicudata.so.50.2
00007fdc4f816000 84K r-x-- libgcc_s-4.8.5-20150702.so.1
00007fdc4f82b000 2044K ----- libgcc_s-4.8.5-20150702.so.1
00007fdc4fa2a000 4K r---- libgcc_s-4.8.5-20150702.so.1
00007fdc4fa2b000 4K rw--- libgcc_s-4.8.5-20150702.so.1
00007fdc4fa2c000 932K r-x-- libstdc++.so.6.0.19
00007fdc4fb15000 2044K ----- libstdc++.so.6.0.19
00007fdc4fd14000 32K r---- libstdc++.so.6.0.19
00007fdc4fd1c000 8K rw--- libstdc++.so.6.0.19
00007fdc4fd1e000 84K rw--- [ anon ]
00007fdc4fd33000 232K r-x-- libnspr4.so
00007fdc4fd6d000 2044K ----- libnspr4.so
00007fdc4ff6c000 4K r---- libnspr4.so
00007fdc4ff6d000 8K rw--- libnspr4.so
00007fdc4ff6f000 8K rw--- [ anon ]
00007fdc4ff71000 16K r-x-- libplc4.so
00007fdc4ff75000 2044K ----- libplc4.so
00007fdc50174000 4K r---- libplc4.so
00007fdc50175000 4K rw--- libplc4.so
00007fdc50176000 12K r-x-- libplds4.so
00007fdc50179000 2044K ----- libplds4.so
00007fdc50378000 4K r---- libplds4.so
00007fdc50379000 4K rw--- libplds4.so
00007fdc5037a000 164K r-x-- libnssutil3.so
00007fdc503a3000 2044K ----- libnssutil3.so
00007fdc505a2000 28K r---- libnssutil3.so
00007fdc505a9000 4K rw--- libnssutil3.so
00007fdc505aa000 1176K r-x-- libnss3.so
00007fdc506d0000 2048K ----- libnss3.so
00007fdc508d0000 20K r---- libnss3.so
00007fdc508d5000 8K rw--- libnss3.so
00007fdc508d7000 8K rw--- [ anon ]
00007fdc508d9000 148K r-x-- libsmime3.so
00007fdc508fe000 2044K ----- libsmime3.so
00007fdc50afd000 12K r---- libsmime3.so
00007fdc50b00000 4K rw--- libsmime3.so
00007fdc50b01000 332K r-x-- libssl3.so
00007fdc50b54000 2048K ----- libssl3.so
00007fdc50d54000 16K r---- libssl3.so
00007fdc50d58000 4K rw--- libssl3.so
00007fdc50d59000 4K rw--- [ anon ]
00007fdc50d5a000 112K r-x-- libsasl2.so.3.0.0
00007fdc50d76000 2044K ----- libsasl2.so.3.0.0
00007fdc50f75000 4K r---- libsasl2.so.3.0.0
00007fdc50f76000 4K rw--- libsasl2.so.3.0.0
00007fdc50f77000 56K r-x-- liblber-2.4.so.2.10.7
00007fdc50f85000 2044K ----- liblber-2.4.so.2.10.7
00007fdc51184000 4K r---- liblber-2.4.so.2.10.7
00007fdc51185000 4K rw--- liblber-2.4.so.2.10.7
00007fdc51186000 88K r-x-- libresolv-2.17.so
00007fdc5119c000 2048K ----- libresolv-2.17.so
00007fdc5139c000 4K r---- libresolv-2.17.so
00007fdc5139d000 4K rw--- libresolv-2.17.so
00007fdc5139e000 8K rw--- [ anon ]
00007fdc513a0000 12K r-x-- libkeyutils.so.1.5
00007fdc513a3000 2044K ----- libkeyutils.so.1.5
00007fdc515a2000 4K r---- libkeyutils.so.1.5
00007fdc515a3000 4K rw--- libkeyutils.so.1.5
00007fdc515a4000 56K r-x-- libkrb5support.so.0.1
00007fdc515b2000 2048K ----- libkrb5support.so.0.1
00007fdc517b2000 4K r---- libkrb5support.so.0.1
00007fdc517b3000 4K rw--- libkrb5support.so.0.1
00007fdc517b4000 196K r-x-- libk5crypto.so.3.1
00007fdc517e5000 2044K ----- libk5crypto.so.3.1
00007fdc519e4000 8K r---- libk5crypto.so.3.1
00007fdc519e6000 4K rw--- libk5crypto.so.3.1
00007fdc519e7000 12K r-x-- libcom_err.so.2.1
00007fdc519ea000 2044K ----- libcom_err.so.2.1
00007fdc51be9000 4K r---- libcom_err.so.2.1
00007fdc51bea000 4K rw--- libcom_err.so.2.1
00007fdc51beb000 868K r-x-- libkrb5.so.3.3
00007fdc51cc4000 2044K ----- libkrb5.so.3.3
00007fdc51ec3000 56K r---- libkrb5.so.3.3
00007fdc51ed1000 12K rw--- libkrb5.so.3.3
00007fdc51ed4000 120K r-x-- libaudit.so.1.0.0
00007fdc51ef2000 2044K ----- libaudit.so.1.0.0
00007fdc520f1000 4K r---- libaudit.so.1.0.0
00007fdc520f2000 4K rw--- libaudit.so.1.0.0
00007fdc520f3000 40K rw--- [ anon ]
00007fdc520fd000 148K r-x-- liblzma.so.5.2.2
00007fdc52122000 2044K ----- liblzma.so.5.2.2
00007fdc52321000 4K r---- liblzma.so.5.2.2
00007fdc52322000 4K rw--- liblzma.so.5.2.2
00007fdc52323000 84K r-x-- libz.so.1.2.7
00007fdc52338000 2044K ----- libz.so.1.2.7
00007fdc52537000 4K r---- libz.so.1.2.7
00007fdc52538000 4K rw--- libz.so.1.2.7
00007fdc52539000 1804K r-x-- libc-2.17.so
00007fdc526fc000 2048K ----- libc-2.17.so
00007fdc528fc000 16K r---- libc-2.17.so
00007fdc52900000 8K rw--- libc-2.17.so
00007fdc52902000 20K rw--- [ anon ]
00007fdc52907000 188K r-x-- libsystemd.so.0.6.0
00007fdc52936000 2048K ----- libsystemd.so.0.6.0
00007fdc52b36000 4K r---- libsystemd.so.0.6.0
00007fdc52b37000 4K rw--- libsystemd.so.0.6.0
00007fdc52b38000 1424K r-x-- libicuuc.so.50.2
00007fdc52c9c000 2048K ----- libicuuc.so.50.2
00007fdc52e9c000 64K r---- libicuuc.so.50.2
00007fdc52eac000 4K rw--- libicuuc.so.50.2
00007fdc52ead000 16K rw--- [ anon ]
00007fdc52eb1000 1984K r-x-- libicui18n.so.50.2
00007fdc530a1000 2048K ----- libicui18n.so.50.2
00007fdc532a1000 48K r---- libicui18n.so.50.2
00007fdc532ad000 8K rw--- libicui18n.so.50.2
00007fdc532af000 4K rw--- [ anon ]
00007fdc532b0000 328K r-x-- libldap-2.4.so.2.10.7
00007fdc53302000 2048K ----- libldap-2.4.so.2.10.7
00007fdc53502000 8K r---- libldap-2.4.so.2.10.7
00007fdc53504000 4K rw--- libldap-2.4.so.2.10.7
00007fdc53505000 1028K r-x-- libm-2.17.so
00007fdc53606000 2044K ----- libm-2.17.so
00007fdc53805000 4K r---- libm-2.17.so
00007fdc53806000 4K rw--- libm-2.17.so
00007fdc53807000 8K r-x-- libdl-2.17.so
00007fdc53809000 2048K ----- libdl-2.17.so
00007fdc53a09000 4K r---- libdl-2.17.so
00007fdc53a0a000 4K rw--- libdl-2.17.so
00007fdc53a0b000 28K r-x-- librt-2.17.so
00007fdc53a12000 2044K ----- librt-2.17.so
00007fdc53c11000 4K r---- librt-2.17.so
00007fdc53c12000 4K rw--- librt-2.17.so
00007fdc53c13000 92K r-x-- libpthread-2.17.so
00007fdc53c2a000 2044K ----- libpthread-2.17.so
00007fdc53e29000 4K r---- libpthread-2.17.so
00007fdc53e2a000 4K rw--- libpthread-2.17.so
00007fdc53e2b000 16K rw--- [ anon ]
00007fdc53e2f000 296K r-x-- libgssapi_krb5.so.2.2
00007fdc53e79000 2048K ----- libgssapi_krb5.so.2.2
00007fdc54079000 4K r---- libgssapi_krb5.so.2.2
00007fdc5407a000 8K rw--- libgssapi_krb5.so.2.2
00007fdc5407c000 2264K r-x-- libcrypto.so.1.0.2k
00007fdc542b2000 2048K ----- libcrypto.so.1.0.2k
00007fdc544b2000 112K r---- libcrypto.so.1.0.2k
00007fdc544ce000 52K rw--- libcrypto.so.1.0.2k
00007fdc544db000 16K rw--- [ anon ]
00007fdc544df000 412K r-x-- libssl.so.1.0.2k
00007fdc54546000 2048K ----- libssl.so.1.0.2k
00007fdc54746000 16K r---- libssl.so.1.0.2k
00007fdc5474a000 28K rw--- libssl.so.1.0.2k
00007fdc54751000 52K r-x-- libpam.so.0.83.1
00007fdc5475e000 2048K ----- libpam.so.0.83.1
00007fdc5495e000 4K r---- libpam.so.0.83.1
00007fdc5495f000 4K rw--- libpam.so.0.83.1
00007fdc54960000 1404K r-x-- libxml2.so.2.9.1
00007fdc54abf000 2044K ----- libxml2.so.2.9.1
00007fdc54cbe000 32K r---- libxml2.so.2.9.1
00007fdc54cc6000 8K rw--- libxml2.so.2.9.1
00007fdc54cc8000 8K rw--- [ anon ]
00007fdc54cca000 80K r-x-- liblz4.so.1.7.5
00007fdc54cde000 2044K ----- liblz4.so.1.7.5
00007fdc54edd000 4K r---- liblz4.so.1.7.5
00007fdc54ede000 4K rw--- liblz4.so.1.7.5
00007fdc54edf000 136K r-x-- ld-2.17.so
00007fdc54f1a000 1588K rw--- [ anon ]
00007fdc550a7000 212K rw-s- PostgreSQL.4009629762
00007fdc550dc000 104K rw--- [ anon ]
00007fdc550fc000 8K r-x-- [ anon ]
00007fdc550fe000 4K rw-s- [ shmid=0x67418000 ]
00007fdc550ff000 4K rw--- [ anon ]
00007fdc55100000 4K r---- ld-2.17.so
00007fdc55101000 4K rw--- ld-2.17.so
00007fdc55102000 4K rw--- [ anon ]
00007fffd382b000 132K rw--- [ stack ]
00007fffd38f0000 8K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
total 12212288K

-bash-4.2$ smem -u
User Count Swap USS PSS RSS
postgres 326 0 10503276 20969859 175615624
-bash-4.2$ smem
^C
PID User Command Swap USS PSS
RSS
27250 postgres postgres: db-prematch: arch 0 224 246
2592
27251 postgres postgres: db-prematch: stat 0 512 538
2716
27253 postgres postgres: db-prematch: logi 0 548 567
3480
27249 postgres postgres: db-prematch: auto 0 664 696
3956
27248 postgres postgres: db-prematch: walw 0 244 878
19296
...27246 postgres postgres: db-prematch: chec 0 88308 851490
10271612
12156 postgres postgres: db-prematch: app_ 0 902776 1400234
9066116
8303 postgres postgres: db-prematch: app_ 0 913768 1411286
9067356
16469 postgres postgres: db-prematch: app_ 0 923668 1417847
9071168
31187 postgres postgres: db-prematch: app_ 0 923024 1425936
9196732
13769 postgres postgres: db-prematch: app_ 0 925820 1437147
9283344
26144 postgres postgres: db-prematch: app_ 0 937440 1437802
9180720
15449 postgres postgres: db-prematch: app_ 0 932776 1440671
9262056
22570 postgres postgres: db-prematch: app_ 0 938680 1447851
9292176
2910 postgres postgres: db-prematch: app_ 0 1039452 1536980
9183668
17048 postgres postgres: db-prematch: app_ 0 1054244 1564436
9425556

-bash-4.2$ smem -w
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 18242380 16741684 1500696
userspace memory 21358228 10746068 10612160
free memory 1433148 1433148 0

-bash-4.2$ smem -m
Map PIDs AVGPSS PSS
/dev/shm/PostgreSQL.4009629762 316 0 0
/usr/lib64/gconv/gconv-modules.cache 18 0 0
/usr/lib64/libattr.so.1.1.0 318 0 0
...
/usr/pgsql-14/lib/libpq.so.5.14 15 27 415
/usr/lib64/libkrb5.so.3.3 319 1 445
/usr/pgsql-14/lib/pg_qualstats.so 318 1 572
/usr/pgsql-14/lib/pg_stat_kcache.so 318 1 572
/usr/lib64/libpython2.7.so.1.0 1 945 945
/usr/pgsql-14/lib/pg_stat_statements.so 318 3 1212
/usr/lib64/libpthread-2.17.so 320 4 1304
/usr/lib64/ld-2.17.so 322 4 1373
/usr/pgsql-14/lib/plpgsql.so 318 4 1376
/usr/lib64/libcrypto.so.1.0.2k 319 4 1499
/usr/lib64/libc-2.17.so 322 9 3179
[stack] 322 26 8498
/usr/pgsql-14/bin/postgres 318 57 18281
/usr/lib64/llvm5.0/lib/libLLVM-5.0.so 10 4098 40984
<anonymous> 322 6557 2111438
[heap] 322 25093 8080218
/dev/zero 316 33834 10691660

Kind Regards Ales Zeleny

>
> If the RssAnon memory is a good indicator, i can then determine the
> backends and dump memory context.
> It'll take me some time since I'm out of office for vacation, but I'll
> manage that somewhat way.
>
> Thanks for all to the hints!
>
> Aleš
>
>>
>> regards
>>
>> --
>> Tomas Vondra
>> EnterpriseDB: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>