Re: Logical replication row filter loses unchanged toasted columns

Thank you all for the discussion. We now have options A to E and
Amit's Option 1 and 2, so let me sort the proposals by where each one
intervenes.

- Publication DDL time: forbid creating the publication (Matthias's
D), or warn (Greg's E).

- UPDATE time on the publisher: reject the UPDATE, per row in
heap_update() (Amit's Option 1, Hou's 0001), or per statement in
CheckCmdReplicaIdentity() (Amit's Option 2, Hou's 0002).

- Decode time on the publisher: error in pgoutput_row_filter() (my A).

- Apply time on the subscriber: error in apply_handle_insert() (Nikhil's patch).

- Make it work instead of erroring: WAL-log the missing values, either
always (my B) or for user-chosen columns (the INCLUDE-like idea
upthread).

- Document only (my C).

The earlier a check runs, the more it prevents and the less it knows.
The DDL and UPDATE time checks fire before anything is written to WAL,
but they have to be conservative. Even Option 1 rejects an UPDATE
whose old and new rows both match the filter, which replicates fine as
a plain UPDATE today. The decode and apply time checks are precise,
they fire exactly when a value is dropped, but by then the value is
gone. Between those two, the apply time error is recoverable with
ALTER SUBSCRIPTION SKIP while the decode time error leaves the slot
stuck, so Nikhil's check supersedes my A.

Given that, the combination I would aim for is:

- All branches (15+): Nikhil's apply time error plus a documentation
note. This is not redundant on master even after an UPDATE time check
lands there, because a master subscriber can replicate from an older
publisher that has no such check.

- master: additionally reject at UPDATE time. I agree with Amit's lean
towards Option 1. Option 2 rejects every UPDATE on a row-filtered
table that merely has a toastable column outside the replica identity,
which is close to D in impact.

- Future: the INCLUDE-like logging in a separate thread, which would
turn the remaining errors into working replication.

I will review Nikhil's patch and Hou's patches later.

Thoughts?

--
Shinya Kato
NTT OSS Center