Protect syscache from bloating with negative cache entries

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Protect syscache from bloating with negative cache entries
Date: 2016-12-19 11:15:05
Message-ID: 20161219.201505.11562604.horiguchi.kyotaro@lab.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

Hello, recently one of my customer stumbled over an immoderate
catcache bloat.

This is a known issue living on the Todo page in the PostgreSQL
wiki.

https://wiki.postgresql.org/wiki/Todo#Cache_Usage
> Fix memory leak caused by negative catcache entries

https://www.postgresql.org/message-id/51C0A1FF.2050404@vmware.com

This patch addresses the two cases of syscache bloat by using
invalidation callback mechanism.

Overview of the patch

The bloat is caused by negative cache entries in catcaches. They
are crucial for performance but it is a problem that there's no
way to remove them. They last for the backends' lifetime.

The first patch provides a means to flush catcache negative
entries, then defines a relcache invalidation callback to flush
negative entries in syscaches for pg_statistic(STATRELATTINH) and
pg_attributes(ATTNAME, ATTNUM). The second patch implements a
syscache invalidation callback so that deletion of a schema
causes a flush for pg_class (RELNAMENSP).

Both of the aboves are not hard-coded and defined in cacheinfo
using additional four members.

Remaining problems

Still, catcache can bloat by repeatedly accessing non-existent
table with unique names in a permanently-living schema but it
seems a bit too artificial (or malicious). Since such negative
entries don't have a trigger to remove, caps are needed to
prevent them from bloating syscaches, but the limits are hardly
seem reasonably determinable.

Defects or disadvantages

This patch scans over whole the target catcache to find negative
entries to remove and it might take a (comparably) long time on a
catcache with so many entries. By the second patch, unrelated
negative caches may be involved in flushing since they are keyd
by hashvalue, not by the exact key values.

The attached files are the following.

1. 0001-Cleanup-negative-cache-of-pg_statistic-when-dropping.patch
Negative entry flushing by relcache invalidation using
relcache invalidation callback.

2. 0002-Cleanup-negative-cache-of-pg_class-when-dropping-a-s.patch
Negative entry flushing by catcache invalidation using
catcache invalidation callback.

3. gen.pl
a test script for STATRELATTINH bloating.

4. gen2.pl
a test script for RELNAMENSP bloating.

3 and 4 are used as the following,

./gen.pl | psql postgres > /dev/null

regards,

--
Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
0001-Cleanup-negative-cache-of-pg_statistic-when-dropping.patch text/x-patch 21.1 KB
0002-Cleanup-negative-cache-of-pg_class-when-dropping-a-s.patch text/x-patch 19.7 KB
unknown_filename text/plain 240 bytes
unknown_filename text/plain 165 bytes

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Etsuro Fujita 2016-12-19 12:05:49 Re: postgres_fdw bug in 9.6
Previous Message Thomas Munro 2016-12-19 11:13:19 Re: Measuring replay lag