Re: shared-memory based stats collector

At Thu, 3 Sep 2020 13:16:59 -0400, Stephen Frost <sfrost(at)snowman(dot)net> wrote in
> Greetings,
>
> * Kyotaro Horiguchi (horikyota(dot)ntt(at)gmail(dot)com) wrote:
> > At Mon, 01 Jun 2020 18:00:01 +0900 (JST), Kyotaro Horiguchi <horikyota(dot)ntt(at)gmail(dot)com> wrote in
> > > Rebased on the current HEAD. 36ac359d36 conflicts with this. Tranche
> >
> > Hmm. This conflicts with 0fd2a79a63. Reabsed on it.
>
> Thanks for working on this and keeping it updated!
>
> I've started taking a look and at least right off...
>
> > >From 4926e50e7635548f86dcd0d36cbf56d168a5d242 Mon Sep 17 00:00:00 2001
> > From: Kyotaro Horiguchi <horikyoga(dot)ntt(at)gmail(dot)com>
> > Date: Mon, 16 Mar 2020 17:15:35 +0900
> > Subject: [PATCH v35 1/7] Use standard crash handler in archiver.
> >
> > The commit 8e19a82640 changed SIGQUIT handler of almost all processes
> > not to run atexit callbacks for safety. Archiver process should behave
> > the same way for the same reason. Exit status changes 1 to 2 but that
> > doesn't make any behavioral change.
>
> Shouldn't this:
>
> a) be back-patched, as the other change was
> b) also include a change to have the stats collector (which I realize is
> removed later on in this patch set, but we're talking about fixing
> existing things..) for the same reason, and because there isn't much
> point in trying to write out the stats after we get a SIGQUIT, since
> we're just going to blow them away again since we're going to go
> through crash recovery..?
>
> Might be good to have a separate thread to address these changes.

+1

> I've looked through (some of) this thread and through the patches also
> and hope to provide a review of the bits that should be targetting v14
> (unlike the above) soon.

Thanks. Current the patch is found to lead to write contention heavier
than the current stats collector when nearly a thousand backend
heavily write to the same table. I need to address that.

- Collect stats via sockets (in the same way as the current implement)
and write/read to/from shared memory.

- Use our own lock-free (maybe) ring-buffer before stats-write enters
lock-waiting mode, then new stats collector(!) process consumes the
queue.

- Some other measures..

Anyway, I'll post a rebased version soon.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center