Re: shared-memory based stats collector

On Fri, Jul 6, 2018 at 10:29 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> I agree that this is definitely something that needs to be considered. I
> took a look some time ago at the same thing, and ran up against exactly that
> one (and at the time did not have time to fix it).
>
> I have not yet had time to look at the downstream suggested handling (UNDO).
> However, I had one other thing from my notes I wanted to mention :)
>
> We should probably consider adding an API to fetch counters that *don't*
> follow these rules, in case it's not needed. When going through files we're
> still stuck at that bottleneck, but if going through shared memory it should
> be possible to make it a lot cheaper by volunteering to "not need that".
>
> We should also consider the ability to fetch stats for a single object,
> which would require no copying of the whole structure at all. I think
> something like this could for example be used for autovacuum rechecks. On
> top of the file based transfer that would help very little, but through
> shared memory it could be a lot lighter weight.

I think we also have to ask ourselves in general whether snapshots of
this data are worth what they cost. I don't think anyone would doubt
that a consistent snapshot of the data is better than an inconsistent
view of the data if the costs were equal. However, if we can avoid a
huge amount of memory usage and complexity on large systems with
hundreds of backends by ditching the snapshot requirement, then we
should ask ourselves how important we think the snapshot behavior
really is.

Note that commit 3cba8999b34 relaxed the synchronization requirements
around GetLockStatusData(). In other words, since 2011, you can no
longer be certain that 'select * from pg_locks' is returning a
perfectly consistent view of the lock status. If this has caused
anybody a major problem, I'm unaware of it. Maybe the same would end
up being true here. The amount of memory we're consuming for this
data may be a bigger problem than minor inconsistencies in the view of
the data would be.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company