Re: shared-memory based stats collector

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
To: robertmhaas(at)gmail(dot)com
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: shared-memory based stats collector
Date: 2018-07-04 08:29:11
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Hello. This is new version fixed windows build.

At Tue, 03 Jul 2018 19:01:44 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> wrote in <20180703(dot)190144(dot)222427588(dot)horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
> Hello. Thanks for the comment.
> At Mon, 2 Jul 2018 14:25:58 -0400, Robert Haas <robertmhaas(at)gmail(dot)com> wrote in <CA+TgmoYQhr30eAcgJCi1v0FhA+3RP1FZVnXqSTLe=6fHy9e5oA(at)mail(dot)gmail(dot)com>
> > Copying the whole hash table kinds of sucks, partly because of the
> > time it will take to copy it, but also because it means that memory
> > usage is still O(nbackends * ntables). Without looking at the patch,
> > I'm guessing that you're doing that because we need a way to show each
> > transaction a consistent snapshot of the data, and I admit that I
> > don't see another obvious way to tackle that problem. Still, it would
> > be nice if we had a better idea.
> The consistency here means "repeatable read" of an object's stats
> entry, not a snapshot covering all objects. We don't need to copy
> all the entries at once following this definition. The attached
> version makes a cache entry only for requested objects.
> Addition to that vacuum doesn't require even repeatable read
> consistency so we don't need to cache the entries at all.
> backend_get_tab_entry now returns an isolated (that means
> not-stored-in-hash) palloc'ed copy without making a local copy in
> the case.
> As the result, this version behaves as the follows.
> - Stats collector stores the results in shared memory.
> - In backend, cache is created only for requested objects and
> lasts for the transaction.
> - Vacuum directly reads the shared stats and doesn't create a
> local copy.
> The non-behavioral difference from the v1 is the follows.
> - snapshot feature of dshash is removed.

This version includes some additional patches. 0003 removes
PG_STAT_TMP_DIR and it affects pg_stat_statements, pg_basebackup
and pg_rewind. Among them pg_stat_statements gets build failure
because it uses the directory to save query texts. 0005 is a new
patch and moves the file to the permanent stat directory. With
this change pg_basebackup and pg_rewind no longer ignore the
query text file.

I haven't explicitly mentioned that, but
dynamic_shared_memory_type = none prevents server from
starting. This patch is not providing a fallback path for the
case. I'm expecting that 'none' will be removed in v12.

- Functionally same with v2. got cosmetic changes.

- Fixed for Windows build.

- Cosmetic changes from v2.

- New patch in v3 of documentation edits.

- New patch of tentative change of pg_stat_statements.

- New patch of tentative change of pg_rewind.


Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
v3-0001-sequential-scan-for-dshash.patch text/x-patch 6.8 KB
v3-0002-Change-stats-collector-to-an-axiliary-process.patch text/x-patch 13.3 KB
v3-0003-dshash-based-stats-collector.patch text/x-patch 86.9 KB
v3-0004-Documentation-update.patch text/x-patch 6.1 KB
v3-0005-Let-pg_stat_statements-not-to-use-PG_STAT_TMP_DIR.patch text/x-patch 1.9 KB
v3-0006-Remove-pg_stat_tmp-exclusion-from-pg_rewind.patch text/x-patch 1.1 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Peter Eisentraut 2018-07-04 08:46:30 Re: CREATE TABLE .. LIKE .. EXCLUDING documentation
Previous Message Kyotaro HORIGUCHI 2018-07-04 08:28:38 Re: [HACKERS] Restricting maximum keep segments by repslots