Re: Logical insert/update/delete WAL records for custom table AMs

On Sun, Oct 31, 2021 at 11:40 PM Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
>
> Attached is a WIP patch to add new WAL records to represent a logical
> insert, update, or delete. These records do not do anything at REDO
> time, they are only processed during logical decoding/replication.
>
> These are intended to be used by a custom table AM, like my columnar
> compression extension[0], which currently supports physical replication
> but can't support logical decoding/replication because decoding is not
> extensible. Using these new logical records would be redundant, making
> inserts/updates/deletes less efficient, but at least logical decoding
> would work (the lack of which is columnar's biggest weakness).
>
> Alternatively, we could support extensible WAL with extensible
> decoding. I also like this approach, but it takes more work for an AM
> like columnar to get that right -- it needs to keep additional state in
> the walsender to track and assemble the compressed columns stored
> across many blocks. It also requires a lot of care, because mistakes
> can get you into serious trouble.
>
> This proposal, for new logical records without WAL extensibility,
> provides a more shallow ramp to get a table AM working (including
> logical replication/decoding) without the need to invest in the WAL
> design. Later, of course I'd like the option for extensible WAL as well
> (to be more efficient), but right now I'd prefer it just worked
> (inefficiently).
>

You have modeled DecodeLogicalInsert based on current DecodeInsert and
it generates the same change message, so not sure how exactly these
new messages will be different from current heap_insert/update/delete
messages? Also, we have code to deal with different types of messages
at various places during decoding, so if they will be different then
we need to probably deal at those places as well.

--
With Regards,
Amit Kapila.