Re: shared-memory based stats collector

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
To: robertmhaas(at)gmail(dot)com
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: shared-memory based stats collector
Date: 2018-07-03 10:01:44
Message-ID: 20180703.190144.222427588.horiguchi.kyotaro@lab.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

Hello. Thanks for the comment.

At Mon, 2 Jul 2018 14:25:58 -0400, Robert Haas <robertmhaas(at)gmail(dot)com> wrote in <CA+TgmoYQhr30eAcgJCi1v0FhA+3RP1FZVnXqSTLe=6fHy9e5oA(at)mail(dot)gmail(dot)com>
> On Fri, Jun 29, 2018 at 4:34 AM, Kyotaro HORIGUCHI
> <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> wrote:
> > Nowadays PostgreSQL has dynamic shared hash (dshash) so we can
> > use this as the main storage of statistics. We can share data
> > without a stress using this.
> >
> > A PoC previously posted tried to use "locally copied" dshash but
> > it doesn't looks fine so I steered to different direction.
> >
> > With this patch dshash can create a local copy based on dynhash.
>
> Copying the whole hash table kinds of sucks, partly because of the
> time it will take to copy it, but also because it means that memory
> usage is still O(nbackends * ntables). Without looking at the patch,
> I'm guessing that you're doing that because we need a way to show each
> transaction a consistent snapshot of the data, and I admit that I
> don't see another obvious way to tackle that problem. Still, it would
> be nice if we had a better idea.

The consistency here means "repeatable read" of an object's stats
entry, not a snapshot covering all objects. We don't need to copy
all the entries at once following this definition. The attached
version makes a cache entry only for requested objects.

Addition to that vacuum doesn't require even repeatable read
consistency so we don't need to cache the entries at all.
backend_get_tab_entry now returns an isolated (that means
not-stored-in-hash) palloc'ed copy without making a local copy in
the case.

As the result, this version behaves as the follows.

- Stats collector stores the results in shared memory.

- In backend, cache is created only for requested objects and
lasts for the transaction.

- Vacuum directly reads the shared stats and doesn't create a
local copy.

The non-behavioral difference from the v1 is the follows.

- snapshot feature of dshash is removed.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
v2-0001-sequential-scan-for-dshash.patch text/x-patch 5.9 KB
v2-0002-Change-stats-collector-to-an-axiliary-process.patch text/x-patch 10.2 KB
v2-0003-dshash-based-stats-collector.patch text/x-patch 87.2 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message ROS Didier 2018-07-03 10:17:13 How to use public key file to encrypt data
Previous Message Michael Paquier 2018-07-03 09:53:43 Re: pgsql: Clarify use of temporary tables within partition trees