Re: shared-memory based stats collector

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
To: andres(at)anarazel(dot)de
Cc: ah(at)cybertec(dot)at, magnus(at)hagander(dot)net, robertmhaas(at)gmail(dot)com, tgl(at)sss(dot)pgh(dot)pa(dot)us, pgsql-hackers(at)postgresql(dot)org
Subject: Re: shared-memory based stats collector
Date: 2018-09-27 13:00:49
Message-ID: 20180927.220049.168546206.horiguchi.kyotaro@lab.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hello. This is a super-PoC of no-UDP stats collector.

At Wed, 26 Sep 2018 09:55:09 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> wrote in <20180926(dot)095509(dot)182252925(dot)horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
> > I don't think either of these is right. I think it's crucial to get rid
> > of the UDP socket, but I think using a shmem queue is the wrong
> > approach. Not just because postgres' shm_mq is single-reader/writer, but
> > also because it's plainly unnecessary. Backends should attempt to
> > update the shared hashtable, but acquire the necessary lock
> > conditionally, and leave the pending updates of the shared hashtable to
> > a later time if they cannot acquire the lock.
>
> Ok, I just intended to avoid reading many bytes from a file and
> thought that writer-side can be resolved later.
>
> Currently locks on the shared stats table is acquired by dshash
> mechanism in a partition-wise manner. The number of the
> partitions is currently fixed to 2^7 = 128, but writes for the
> same table confilicts each other regardless of the number of
> partitions. As the first step, I'm going to add
> conditional-locking capability to dsahsh_find_or_insert and each
> backend holds a queue of its pending updates.

I don't have more time 'til next monday so this is just a PoC
(sorry..).

- 0001 to 0006 is rebased version of v4.
- 0007 adds conditional locking to dshash

- 0008 is the no-UDP stats collector.

If required lock is not acquired for some stats items, report
funcions immediately return after storing the values locally. The
stored values are merged with later calls. Explicitly calling
pgstat_cleanup_pending_stat() at a convenient timing tries to
apply the pending values, but the function is not called anywhere
for now.

stats collector process is used only to save and load saved stats
files and create shared memory for stats. I'm going to remove
stats collector.

I'll continue working this way.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
v5-0008-Ultra-PoC-of-full-shared-memory-stats-collector.patch text/x-patch 92.5 KB
v5-0007-Add-conditional-lock-feature-to-dshash.patch text/x-patch 4.1 KB
v5-0006-Remove-pg_stat_tmp-exclusion-from-pg_rewind.patch text/x-patch 1.1 KB
v5-0005-Let-pg_stat_statements-not-to-use-PG_STAT_TMP_DIR.patch text/x-patch 1.9 KB
v5-0004-Documentation-update.patch text/x-patch 6.1 KB
v5-0003-dshash-based-stats-collector.patch text/x-patch 93.7 KB
v5-0002-Change-stats-collector-to-an-axiliary-process.patch text/x-patch 13.3 KB
v5-0001-sequential-scan-for-dshash.patch text/x-patch 6.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Dean Rasheed 2018-09-27 13:22:39 Re: BUG #15307: Low numerical precision of (Co-) Variance
Previous Message Michael Paquier 2018-09-27 12:51:55 Re: Problem while setting the fpw with SIGHUP