Re: shared-memory based stats collector

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
To: andres(at)anarazel(dot)de
Cc: magnus(at)hagander(dot)net, robertmhaas(at)gmail(dot)com, tgl(at)sss(dot)pgh(dot)pa(dot)us, pgsql-hackers(at)postgresql(dot)org
Subject: Re: shared-memory based stats collector
Date: 2018-07-10 12:07:40
Message-ID: 20180710.210740.66209890.horiguchi.kyotaro@lab.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

Hello. Thanks for the opinions.

At Fri, 6 Jul 2018 13:10:36 -0700, Andres Freund <andres(at)anarazel(dot)de> wrote in <20180706201036(dot)awheoi6tk556x6aj(at)alap3(dot)anarazel(dot)de>
> Hi,
>
> On 2018-07-06 22:03:12 +0200, Magnus Hagander wrote:
> > *If* we can provide the snapshots view of them without too much overhead I
> > think it's worth looking into that while *also* proviiding a lower overhead
> > interface for those that don't care about it.
>
> I don't see how that's possible without adding significant amounts of
> complexity and probably memory / cpu overhead. The current stats already
> are quite inconsistent (often outdated, partially updated, messages
> dropped when busy) - I don't see what we really gain by building
> something MVCC like in the "new" stats subsystem.
>
>
> > If it ends up that keeping the snapshots become too much overhead in either
> > in performance or code-maintenance, then I agree can probably drop that.
> > But we should at least properly investigate the cost.
>
> I don't think it's worthwhile to more than think a bit about it. There's
> fairly obvious tradeoffs in complexity here. Trying to get there seems
> like a good way to make the feature too big.

Agreed.

Well, if we allow to lose consistency in some extent for improved
performance and smaller footprint, relaxing the consistency of
database stats can reduce footprint further especially on a
cluster with so many databases. Backends are interested only in
the residing database and vacuum doesn't cache stats at all. A
possible problem is vacuum and stats collector can go into a race
condition. I'm not sure but I suppose it is not worse than being
involved in an IO congestion.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
v4-0001-sequential-scan-for-dshash.patch text/x-patch 6.8 KB
v4-0002-Change-stats-collector-to-an-axiliary-process.patch text/x-patch 13.3 KB
v4-0003-dshash-based-stats-collector.patch text/x-patch 93.7 KB
v4-0004-Documentation-update.patch text/x-patch 6.1 KB
v4-0005-Let-pg_stat_statements-not-to-use-PG_STAT_TMP_DIR.patch text/x-patch 1.9 KB
v4-0006-Remove-pg_stat_tmp-exclusion-from-pg_rewind.patch text/x-patch 1.1 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Amit Kapila 2018-07-10 12:49:52 Re: EXPLAIN of Parallel Append
Previous Message Heikki Linnakangas 2018-07-10 11:39:38 Re: Excessive PostmasterIsAlive calls slow down WAL redo