Re: shared-memory based stats collector

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
To: tomas(dot)vondra(at)2ndquadrant(dot)com
Cc: alvherre(at)2ndquadrant(dot)com, andres(at)anarazel(dot)de, ah(at)cybertec(dot)at, magnus(at)hagander(dot)net, robertmhaas(at)gmail(dot)com, tgl(at)sss(dot)pgh(dot)pa(dot)us, pgsql-hackers(at)postgresql(dot)org
Subject: Re: shared-memory based stats collector
Date: 2019-01-22 06:48:02
Message-ID: 20190122.154802.212542603.horiguchi.kyotaro@lab.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

Hello.

At Mon, 21 Jan 2019 21:19:07 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> wrote in <20190121(dot)211907(dot)59625409(dot)horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
> I'll reconsider the referer side of the stats.

The most significant cause of the slowdown is repeated search for
non-existent entries both on local and shared hash each time.

Negative cache in addition to cache expiration interval
eliminates the slowdown.

1000 times repetition with -O2 binary:

master : 124.99 tps
patched: 125.48 tps (+0.4%)

> I didn't merge the suggested two pairs of commits. I'll do that
> after adressing the slowdown issue.

I agree to commiting 0001-0003 separately. In the attached patch
set, old 0004+0006 are merged as 0004 and old 0005+0007 are
merged as new 0005.

Changed cacheing policy.
Expired at every xact end
-> Keeps at least for PGSTAT_STAT_MIN_INTERVAL (500ms).

Added negative cache feature (snapshot_statentry).

Improved separation between pgstat and bestat (separated AtEOXact_* functions).

Fixed doubious memory context usage.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
v12-0001-sequential-scan-for-dshash.patch text/x-patch 10.6 KB
v12-0002-Add-conditional-lock-feature-to-dshash.patch text/x-patch 5.0 KB
v12-0003-Make-archiver-process-an-auxiliary-process.patch text/x-patch 12.0 KB
v12-0004-Shared-memory-based-stats-collector.patch text/x-patch 455.3 KB
v12-0005-Remove-the-GUC-stats_temp_directory.patch text/x-patch 9.9 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tsunakawa, Takayuki 2019-01-22 06:57:22 RE: Thread-unsafe coding in ecpg
Previous Message Tsunakawa, Takayuki 2019-01-22 06:36:18 RE: Protect syscache from bloating with negative cache entries