Re: Protect syscache from bloating with negative cache entries

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
To: tsunakawa(dot)takay(at)jp(dot)fujitsu(dot)com
Cc: alvherre(at)2ndquadrant(dot)com, tomas(dot)vondra(at)2ndquadrant(dot)com, bruce(at)momjian(dot)us, andres(at)anarazel(dot)de, robertmhaas(at)gmail(dot)com, tgl(at)sss(dot)pgh(dot)pa(dot)us, ideriha(dot)takeshi(at)jp(dot)fujitsu(dot)com, pgsql-hackers(at)lists(dot)postgresql(dot)org, michael(dot)paquier(at)gmail(dot)com, david(at)pgmasters(dot)net, craig(at)2ndquadrant(dot)com
Subject: Re: Protect syscache from bloating with negative cache entries
Date: 2019-02-13 06:31:14
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

At Tue, 12 Feb 2019 20:36:28 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> wrote in <20190212(dot)203628(dot)118792892(dot)horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
> > (4)
> > + hash_size = cp->cc_nbuckets * sizeof(dlist_head);
> > + tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len;
> > + tupsize = sizeof(CatCTup);
> >
> > GetMemoryChunkSpace() should be used to include the memory context overhead. That's what the files in src/backend/utils/sort/ do.
> Thanks. Done. Include bucket and cache header part but still
> excluding clist. Renamed from tupsize to memusage.

It is too complex as I was afraid. The indirect calls causes
siginicant degradation. (Anyway the previous code was bogus in
that it passes CACHELINEALIGN'ed pointer to get_chunk_size..)

Instead, I added an accounting(?) interface function.

| MemoryContextGettConsumption(MemoryContext cxt);

The API returns the current consumption in this memory
context. This allows "real" memory accounting almost without

(1) New patch v15-0002 adds accounting feature to MemoryContext.
(It adds this feature only to AllocSet, if this is acceptable
it can be extended to other allocators.)

(2) Another new patch v15-0005 on top of previous design of
limit-by-number-of-a-cache feature converts it to
limit-by-size-on-all-caches feature, which I think is
Tsunakawa-san wanted.

As far as I can see no significant degradation is found in usual
(as long as pruning doesn't happen) code paths.

About the new global-size based evicition(2), cache entry
creation becomes slow after the total size reached to the limit
since every one new entry evicts one or more old (=
not-recently-used) entries. Because of not needing knbos for each
cache, it become far realistic. So I added documentation of
"catalog_cache_max_size" in 0005.

About the age-based eviction, the bulk eviction seems to take a a
bit long time but it happnes instead of hash resizing so the user
doesn't observe additional slowdown. On the contrary the pruning
can avoid rehashing scanning the whole cache. I think it is the
gain seen in the Tomas' experiment.


Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
v15-0001-Add-dlist_move_tail.patch text/x-patch 1.2 KB
v15-0002-Memory-consumption-report-reature-of-memorycontext.patch text/x-patch 4.2 KB
v15-0003-Remove-entries-that-haven-t-been-used-for-a-certain-.patch text/x-patch 26.1 KB
v15-0004-Syscache-usage-tracking-feature.patch text/x-patch 35.6 KB
v15-0005-Global-LRU-based-cache-pruning.patch text/x-patch 11.5 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Surafel Temesgen 2019-02-13 06:36:46 Re: pg_dump multi VALUES INSERT
Previous Message Jamison, Kirk 2019-02-13 05:48:28 RE: Cache relation sizes?