Re: Tepid: selective index updates for heap relations

From: Greg Burd <greg(at)burd(dot)me>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Tepid: selective index updates for heap relations
In-Reply-To: <af02b486-6adc-494c-a357-1dd72f655dcf(at)app(dot)fastmail(dot)com>
References: <af02b486-6adc-494c-a357-1dd72f655dcf(at)app(dot)fastmail(dot)com>

Attached is v71, rebased on master through f58ab18a.

I took the time to split the table/index AM contract and executor
changes out from the heap implementation. This makes the series longer,
but should make it easier to review the interface without first
reviewing the other changes for Tepid.

In thinking this through I might pull out the locator contract into
a new thread to discuss separately along with the other changes to
core, but for now they live here.

Each of these have their own patch near the beginning so that we can
discuss them separately:

* BitmapAnd now learns the table relation from each bitmap index scan,
including scans nested under BitmapOr. For a group where entries for
the same row can name different slots (as can happen with Tepid due to
mid-chain references in indexes), the AM's callback requests a union
and recheck instead of an exact intersection. The locator contract
also has an optional BitmapOr recheck hook.

* The locator descriptor states width, maximum offset, locator stability
and whether an UPDATE preserves the old version. A stable locator does
not imply unchanged contents. The LOCKED_VERSION flags distinguish a
snapshot-visible version from one the command has already locked;
retargeted tells callers when they must re-evaluate a newer version.

* For AMs that overwrite rows (none of which exist in core today, but
many have experimented with this including me in the UNDO[1] proposal
with the FLUX table AM), the executor preserves OLD before the
write. AFTER ROW events can carry OLD and event-time NEW, with spill
and subtransaction cleanup. Foreign-key checks and unique rechecks
still fetch the current version. The contract requires a deleted row
to remain fetchable with SnapshotAny until the deleting transaction
ends.

* Index creation and rebuilds check locator width. The amcanvarlocator
capability remains, but this does not add variable-width transport to
the existing ItemPointerData interfaces. max_offset also bounds the
planner's bitmap paths and GIN's supported offsets.

In the v68 patch set I overloaded the meaning of bit 14 in a TID as a
signal used during BitmapAnd, I never felt that was a good idea and it
was that struggle that forced my hand to either a) give up or b) write
the locator contract. I did (b), but that meant that I needed somewhere
else to hide the hint I'd burried in bit 14 before. In this series I've
done that via the visibility map and I feel a bit less dirty, but I do
have a sense of guilt for doubling its size.

I moved heap's locator-split visibility-map bit and pg_upgrade rewrite
into the heap portion. This widens the visibility map from two to four
bits per heap block, as I said before doubling its size, and requires
rewriting old _vm forks during upgrade which isn't great, but we're
living in a world of not-great trade-offs if we want to solve this so
help me find a better trade-off, get comfortable with this one, or we
put a pin in this idea alltogether and call it a day.

Heap's update and prune/vacuum changes now land together, so no
intermediate patch creates chains that pruning cannot handle. I also
fixed index-only-scan identification in the common scan constructor so
btree retains its leaf pin for genuine index-only scans.

The patch order is:
01-07 Locator contract, version naming, triggers and bitmap hooks
08 Capturing/codifying the existing HOT behavior in tests
09-12 Attribute comparison and generic selective index maintenance
13-15 Heap format, visibility map, update and prune/vacuum support
16-18 Statistics, amcheck and logical-replication apply option
19-20 [DO NOT MERGE] Test AMs and benchmark harness

The test AMs exercise contract properties that heap does not normally
use. They do not validate LOCKED_VERSION semantics for a real in-place
AM; they still use heap's lock and update implementation.

I need to re-benchmark this version and also measure the impact on the
visibility map.

There remains many things that need to be quantified, reviewed, and
deemed "acceptable trade-offs" including:
* the visibility map 2->4 bits
* rewriting old _vm forks during upgrade
* the overhead of additional tuples in longer chains on heap pages
* making it harder to declare a heap page all visible
* the negative impact of 01-07 patches on performance
* the positive impact of 09-18 on performance, table and index bloat
* I'm surely missing something...

At this point, I'd particularly appreciate review of the locator
contract, the rest of the 01-07 changes, and the visibility-map
trade-off. TM_FailureData still exposes cmax, xmax and ctid to callers
outside heap; I have left abstracting those fields for follow-up work
(because I think they fall into the same bucket of "heap-specific leak
into the surrounding code" or "leaky abstraction" or whatever you'd like
to call it) rather than adding it to this (ever growing, now 20 patches)
series.

best.

-greg