Re: Implementing Incremental View Maintenance

From: Yugo NAGATA <nagata(at)sraoss(dot)co(dot)jp>
To: Yugo NAGATA <nagata(at)sraoss(dot)co(dot)jp>
Cc: vignesh C <vignesh21(at)gmail(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Andrew Dunstan <andrew(at)dunslane(dot)net>, Andy Fan <zhihui(dot)fan1213(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, Tatsuo Ishii <ishii(at)sraoss(dot)co(dot)jp>, Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
Subject: Re: Implementing Incremental View Maintenance
Date: 2021-08-02 06:28:34
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Hi hackers,

On Mon, 19 Jul 2021 09:24:30 +0900
Yugo NAGATA <nagata(at)sraoss(dot)co(dot)jp> wrote:

> On Wed, 14 Jul 2021 21:22:37 +0530
> vignesh C <vignesh21(at)gmail(dot)com> wrote:

> > The patch does not apply on Head anymore, could you rebase and post a
> > patch. I'm changing the status to "Waiting for Author".
> Ok. I'll update the patch in a few days.

Attached is the latest patch set to add support for Incremental
Materialized View Maintenance (IVM)

The patches are rebased to the master and also revised with some
code cleaning.

IVM is a way to make materialized views up-to-date in which only
incremental changes are computed and applied on views rather than
recomputing the contents from scratch as REFRESH MATERIALIZED VIEW
does. IVM can update materialized views more efficiently
than recomputation when only small part of the view need updates.

The patch set implements a feature so that materialized views could be
updated automatically and immediately when a base table is modified.

Currently, our IVM implementation supports views which could contain
tuple duplicates whose definition includes:

- inner and outer joins including self-join
- some built-in aggregate functions (count, sum, agv, min, and max)
- a part of subqueries
-- simple subqueries in FROM clause
-- EXISTS subqueries in WHERE clause
- CTEs

We hope the IVM feature would be adopted into pg15. However, the size of
patch set has grown too large through supporting above features. Therefore,
I think it is better to consider only a part of these features for the first
release. Especially, I would like propose the following features for pg15.

- inner joins including self-join
- DISTINCT and views with tuple duplicates
- some built-in aggregate functions (count, sum, agv, min, and max)

By omitting outer-join, sub-queries, and CTE features, the patch size becomes
less than half. I hope this will make a bit easer to review the IVM patch set.

Here is a list of separated patches.

- 0001: Add a new syntax:
- 0002: Add a new column relisivm to pg_class
- 0003: Add new deptype option 'm' in pg_depend
- 0004: Change trigger.c to allow to prolong life span of tupestores
containing Transition Tables generated via AFTER trigger
- 0005: Add IVM supprot for pg_dump
- 0006: Add IVM support for psql
- 0007: Add the basic IVM future:
This supports inner joins, DISTINCT, and tuple duplicates.
- 0008: Add aggregates (count, sum, avg, min, max) support for IVM
- 0009: Add regression tests for IVM
- 0010: Add documentation for IVM

We could split the patch furthermore if this would make reviews much easer.
For example, I think 0007 could be split into the more basic part and the part
for handling tuple duplicates. Moreover, 0008 could be split into "min/max"
and other aggregates because handling min/max is a bit more complicated than

I also attached IVM_extra.tar.gz that contains patches for sub-quereis,
outer-join, CTE support, just for your information.

Yugo Nagata

Yugo NAGATA <nagata(at)sraoss(dot)co(dot)jp>

Yugo NAGATA <nagata(at)sraoss(dot)co(dot)jp>

Attachment Content-Type Size
IVM_extra.tar.gz application/gzip 30.9 KB
v23-0001-Add-a-syntax-to-create-Incrementally-Maintainabl.patch text/x-diff 5.0 KB
v23-0002-Add-relisivm-column-to-pg_class-system-catalog.patch text/x-diff 4.7 KB
v23-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch text/x-diff 1.7 KB
v23-0004-Allow-to-prolong-life-span-of-transition-tables-.patch text/x-diff 5.6 KB
v23-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch text/x-diff 7.5 KB
v23-0006-Add-Incremental-View-Maintenance-support-to-psql.patch text/x-diff 4.8 KB
v23-0007-Add-Incremental-View-Maintenance-support.patch text/x-diff 89.3 KB
v23-0008-Add-aggregates-support-in-IVM.patch text/x-diff 54.7 KB
v23-0009-Add-regression-tests-for-Incremental-View-Mainte.patch text/x-diff 44.3 KB
v23-0010-Add-documentations-about-Incremental-View-Mainte.patch text/x-diff 25.0 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Amit Langote 2021-08-02 06:29:37 Re: Skip partition tuple routing with constant partition key
Previous Message Greg Nancarrow 2021-08-02 06:04:18 Re: Parallel Inserts (WAS: [bug?] Missed parallel safety checks..)