Re: WAL usage calculation patch

From: Julien Rouhaud <rjuju123(at)gmail(dot)com>
To: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
Cc: Fujii Masao <masao(dot)fujii(at)oss(dot)nttdata(dot)com>, Kirill Bychik <kirill(dot)bychik(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>, Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
Subject: Re: WAL usage calculation patch
Date: 2020-03-30 12:43:56
Message-ID: 20200330124356.GE79261@nol
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Mon, Mar 30, 2020 at 03:52:38PM +0530, Amit Kapila wrote:
> On Sun, Mar 29, 2020 at 5:49 PM Julien Rouhaud <rjuju123(at)gmail(dot)com> wrote:
> >
>
> @@ -1249,6 +1250,16 @@ XLogInsertRecord(XLogRecData *rdata,
> ProcLastRecPtr = StartPos;
> XactLastRecEnd = EndPos;
>
> + /* Provide WAL update data to the instrumentation */
> + if (inserted)
> + {
> + pgWalUsage.wal_bytes += rechdr->xl_tot_len;
> + if (doPageWrites && fpw_lsn <= RedoRecPtr)
> + pgWalUsage.wal_fpw_records++;
> + else
> + pgWalUsage.wal_records++;
> + }
> +
>
> I think the above code has multiple problems. (a) fpw_lsn can be
> InvalidXLogRecPtr and still there could be full-page image (for ex.
> when REGBUF_FORCE_IMAGE flag for buffer is set). (b) There could be
> multiple FPW records while inserting a record; consider when there are
> multiple registered buffers. I think the right place to figure this
> out is XLogRecordAssemble. (c) There are cases when we also attach the
> record data even when we decide to write FPW (cf. REGBUF_KEEP_DATA),
> so we might want to increment wal_fpw_records and wal_records for such
> cases.
>
> I think the right place to compute this information is
> XLogRecordAssemble even though we update it at the place where you
> have it in the patch. You can probably compute that in local
> variables and then transfer to pgWalUsage in XLogInsertRecord. I am
> fine if you can think of some other way but the current patch doesn't
> seem correct to me.

My previous approach was indeed totally broken. v8 attached which hopefully
will be ok.

Attachment Content-Type Size
v8-0001-Add-infrastructure-to-track-WAL-usage.patch text/plain 17.4 KB
v8-0002-Add-option-to-report-WAL-usage-in-EXPLAIN-and-aut.patch text/plain 11.3 KB
v8-0003-Keep-track-of-WAL-usage-in-pg_stat_statements.patch text/plain 21.9 KB
v8-0004-Expose-WAL-usage-counters-in-verbose-auto-vacuum-.patch text/plain 3.0 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Surafel Temesgen 2020-03-30 12:46:13 Re: Conflict handling for COPY FROM
Previous Message John Naylor 2020-03-30 12:30:32 Re: truncating timestamps on arbitrary intervals