Re: WAL usage calculation patch

From: Julien Rouhaud <rjuju123(at)gmail(dot)com>
To: Kirill Bychik <kirill(dot)bychik(at)gmail(dot)com>
Cc: PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>, Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
Subject: Re: WAL usage calculation patch
Date: 2020-03-17 15:31:36
Message-ID: 20200317153136.GA63950@nol
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Sun, Mar 15, 2020 at 09:52:18PM +0300, Kirill Bychik wrote:
> > > > On Thu, Mar 5, 2020 at 8:55 PM Kirill Bychik <kirill(dot)bychik(at)gmail(dot)com> wrote:
> After extensive thinking and some code diving, I did not manage to
> come up with a sane idea on how to expose data about autovacuum WAL
> usage. Must be the flu.
> > > Anyways, I believe this change could be bigger than FPI. I propose to
> > > plan a separate patch for it, or even add it to the TODO after the
> > > core patch of wal usage is merged.
> >
> > Just in case, if the problem is a lack of time, I'd be happy to help
> > on that if needed. Otherwise, I'll definitely not try to block any
> > progress for the feature as proposed.
> Please feel free to work on any extension of this patch idea. I lack
> both time and knowledge to do it all by myself.

I'm adding a 3rd patch on top of yours to expose the new WAL counters in
pg_stat_database, for vacuum and autovacuum. I'm not really enthiusiastic with
this approach but I didn't find better, and maybe this will raise some better
ideas. The only sure thing is that we're not going to add a bunch of new
fields in pg_stat_all_tables anyway.

We can also drop this 3rd patch entirely if no one's happy about it without
impacting the first two.

> > > Please expect a new patch version next week, with FPI counters added.
> Please find attached patch version 003, with FP writes and minor
> corrections. Hope i use attachment versioning as expected in this
> group :)


> Test had been reworked, and I believe it should be stable now, the
> part which checks WAL is written and there is a correlation between
> affected rows and WAL records. I still have no idea how to test
> full-page writes against regular updates, it seems very unstable.
> Please share ideas if any.

I just reviewed the patches, and it globally looks good to me. The way to
detect full page images looks sensible, but I'm really not familiar with that
code so additional review would be useful.

I noticed that the new wal_write_fp_records field in pg_stat_statements wasn't
used in the test. Since I have to add all the patches to make the cfbot happy,
I slightly adapted the tests to reference the fp column too. There was also a
minor issue in the documentation, as wal_records and wal_bytes were copy/pasted
twice while wal_write_fp_records wasn't documented, so I also changed it.

Let me know if you're ok with those changes.

Attachment Content-Type Size
v4-0001-Track-WAL-usage.patch text/plain 13.3 KB
v4-0002-Keep-track-of-WAL-usage-in-pg_stat_statements.patch text/plain 22.6 KB
v4-0003-Keep-track-of-auto-vacuum-WAL-usage-in-pg_stat_da.patch text/plain 13.1 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Tomas Vondra 2020-03-17 15:37:06 Re: PATCH: add support for IN and @> in functional-dependency statistics use
Previous Message Justin Pryzby 2020-03-17 15:24:09 Re: Berserk Autovacuum (let's save next Mandrill)