Re: [PATCH v1] Generic Datum Encoding and Compression Contexts (pg_encoding_type, pg_encoding_context)

Hi

I'm writing this email to express my interest in reviewing your patch, sure
I'll review it once I'm free from my chores.

With Regards
Osama Abdul Qader

On Sun, 18 Oct 2026 at 22:19, Hannu Krosing <hannuk(at)google(dot)com> wrote:

> Hi hackers,
>
> Attached is a patch introducing a generic, extensible Datum Encoding and
> Compression Context subsystem for PostgreSQL (`pg_encoding_type` and
> `pg_encoding_context`).
>
> 1. Motivation
> =============
>
> Currently, column-level compression in PostgreSQL is controlled by the
> single
> `char` column `pg_attribute.attcompression` ('p' for pglz, 'l' for lz4) and
> operates solely as a stateless, parameterless byte-stream transformation.
>
> In practice, many high-value storage and analytical workloads benefit from
> stateful, pre-trained, or schema-aware encoding metadata stored in system
> catalogs and cached in backend memory:
>
> - Dictionary-based compression (e.g., Zstd pre-trained dictionaries for
> JSONB
> or log payloads, FSST symbol tables for short strings) where per-datum
> compression ratios improve dramatically when a shared dictionary is
> factored out of individual tuples.
> - Vector quantization and entropy coding (e.g., scalar quantization
> min/max
> calibration bounds, tANS / asymmetric numeral systems tables, or custom
> codebooks) where a column or type modifier references a trained
> context.
> - Schema-aware binary serialization (e.g., Avro or Protobuf schemas) and
> multi-stage pipelines (e.g., schema encoding followed by dictionary
> compression).
> - Online dictionary / schema evolution without a full table rewrite:
> when a
> better dictionary or new schema version is trained, new writes should
> use
> the new context while historical tuples remain self-describing and
> readable
> using the context with which they were written.
> - Multi-variant fallback within a single column ("Context Groups"): a
> column
> may prefer a specialized encoding variant (e.g., a restricted hex/UUID
> alphabet or a strict schema) that falls back automatically to a more
> general variant if a particular input datum cannot be represented by
> the
> stricter variant, using only a 1-byte (`uint8`) variant selector in the
> encoded payload.
>
>
> 2. Overview of the Design
> =========================
>
> 2.1 System Catalogs
> -------------------
>
> * `pg_encoding_type` (OID 9440):
> Registers pluggable encoding engines (built-in or extension-provided):
> - `oid` (`oid`)
> - `enctypname` (`name`), `enctypnamespace` (`oid`), `enctypowner`
> (`oid`)
> - `enctypkind` (`char`):
> 'd' = dictionary, 'c' = compression, 'q' = quantization, 'e' =
> custom
> - `enctyptype` (`text`): engine identifier (e.g., "pglz",
> "zstd", "xor_lut")
> - `enctypencfunc` (`regproc`): `enc_fn(internal, anyelement) ->
> bytea`
> - `enctypdecfunc` (`regproc`): `dec_fn(internal, bytea) ->
> anyelement`
> - `enctypexpfunc` (`regproc`, optional): `exp_fn(oid, text) -> json`
> - `enctypoptions` (`text[]`): default key=value options
>
> * `pg_encoding_context` (OID 9431):
> Stores trained or configured encoding context instances:
> - `oid` (`oid`)
> - `ectxname` (`name`), `ectxnamespace` (`oid`), `ectxowner` (`oid`)
> - `ectxtypid` (`oid` -> `pg_encoding_type.oid`)
> - `ectxtargettypid` (`oid` -> `pg_type.oid`, or 0 for
> polymorphic/any)
> - `ectxwriteok` (`bool`, default `true`): whether this context may be
> used for encoding new data; setting `ectxwriteok = false` retires a
> context for new writes while keeping it active for decoding
> existing
> tuples.
> - `ectxgroup` (`oid` -> `pg_encoding_context.oid`): context group OID
> stored in `pg_attribute.attencoding_context`. Always equals `oid`
> for
> standalone contexts and group leaders, and equals the group
> leader's
> `oid` for additional variants in the group.
> - `ectxvariant` (`uint8`, `0..255`, default `0`): 1-byte variant
> selector
> within `ectxgroup`. `0` designates a standalone context
> (`ectxgroup = oid`,
> variants not supported). Non-zero values (`1..255`) identify and
> order
> variants within a group.
> - `ectxlabel` (`text`): human-readable version/schema label
> - `ectxenctab` (`bytea`), `ectxdectab` (`bytea`): raw encoding and
> decoding dictionaries/tables (TOAST-able)
> - `ectxoptions` (`text[]`): per-context key=value options
>
> * `pg_attribute.attencoding_context` (`oid`, default `0`):
> Binds a column to an encoding context (or context group leader) for new
> writes, tracked via `pg_depend` (`DEPENDENCY_NORMAL`).
>
>
> 2.2 Context Groups (`ectxgroup` / `ectxvariant`) and Lazy Fallback
> ------------------------------------------------------------------
>
> When a column can be encoded by multiple related contexts (for example, a
> high-density restricted-alphabet encoder as primary variant `1` and a
> general
> compressor as fallback variant `2`), storing a 4-byte OID per inline value
> is
> unnecessary overhead when a 1-byte (`uint8`) variant indicator suffices.
>
> - `pg_attribute.attencoding_context` stores the 4-byte `ectxgroup` OID.
> - Each member of a group has a unique `ectxvariant` in `1..255`.
> - When encoding with a group context,
> `EncodingContextEncodeWithEffectiveOid`
> starts at the lowest `ectxvariant` in the group. If a variant has
> `ectxwriteok = false` or its encode handler declines/fails to encode
> the
> input datum, execution automatically falls back to the context with the
> same `ectxgroup` and the next higher `ectxvariant`, returning both the
> encoded `bytea` and the effective context OID (`*used_context_id`) that
> succeeded.
> - In memory, `EncodingContextData.fallback` is initialized to
> `ECTX_UNRESOLVED` (`(EncodingContextData *)(uintptr_t) 1`) when
> `variant > 0` (and `NULL` when `variant == 0`), and is lazily resolved
> on
> first fallback via `EncodingContextGetFallback()` by scanning the
> group for
> the smallest `ectxvariant > current->variant`.
>
>
> 2.3 Backend Cache & Invalidation
> --------------------------------
>
> `LookupEncodingContext(Oid)` and
> `LookupEncodingContextByGroupVariant(Oid, uint8)`
> materialize `EncodingContextData` entries into a session-local `dynahash`
> table
> allocated inside `EncodingContextMemoryContext` (a child of
> `CacheMemoryContext`).
>
> - Each cached entry has its own child `MemoryContext` so individual
> entries
> can be freed cleanly without leaking or resetting sibling contexts.
> - `CacheRegisterSyscacheCallback` hooks on `ENCODINGCONTEXTOID` and
> `ENCODINGTYPEOID` invalidate cached entries on `ALTER` or `DROP`, and
> reset
> resolved `fallback` pointers back to `ECTX_UNRESOLVED` so group
> membership
> changes take effect immediately across backends.
> - Active encode/decode calls pin entries via `ectx->refcount`; if an
> entry is
> invalidated while pinned on the call stack, destruction is deferred
> until
> the outermost caller releases its pin.
>
>
> 2.4 Common Name / Numeric OID Resolution
> ----------------------------------------
>
> To provide uniform context resolution across core SQL functions, DDL, and
> extension `typmodin` handlers (such as `sqvector(dim, 'ctx_name')` or
> `sqvector(dim, 16385)`):
>
> - C API: `Oid ResolveEncodingContextOid(const char *name_or_oid,
> bool missing_ok)`
> - SQL function: `pg_resolve_encoding_context_oid(name_or_oid text,
> missing_ok bool DEFAULT false) RETURNS oid`
>
> If the input string consists of decimal digits corresponding to an existing
> `pg_encoding_context.oid`, that OID is returned directly; otherwise the
> string
> is parsed as an identifier (supporting schema-qualified `schema.name` or
> `search_path` resolution).
>
>
> 2.5 SQL Functions & Built-in Reference Handlers
> -----------------------------------------------
>
> This patch provides SQL-callable functions for managing, testing, and
> inspecting encoding types and contexts:
>
> - `pg_create_encoding_type`, `pg_drop_encoding_type`
> - `pg_create_encoding_context`, `pg_alter_encoding_context`,
> `pg_drop_encoding_context`
> - `pg_set_column_encoding_context`
> - `pg_resolve_encoding_context_oid`
> - `pg_explain_encoding_context(context_id oid, direction text
> DEFAULT 'encoding') -> json`
> - `pg_encode_datum(context_id oid, val anyelement) -> bytea`
> - `pg_decode_datum(context_id oid, payload bytea, target_type
> anyelement) -> anyelement`
> - `pg_column_raw_encoding(val anyelement) -> bytea`
>
> And two built-in reference encoding engines:
> - `pg_bytea_dict_encode` / `pg_bytea_dict_decode` /
> `pg_bytea_dict_explain`
> (supporting both XOR dictionary transformations and restricted-alphabet
> lookup tables for testing multi-variant group fallback)
> - `pg_pglz_context_encode` / `pg_pglz_context_decode` /
> `pg_pglz_context_explain`
>
>
> 3. Patch Roadmap
> ================
>
> To keep review manageable, this work is structured as a 3-part series:
>
> - Patch 0001 (attached): Core `pg_encoding_type` and
> `pg_encoding_context`
> catalogs, `pg_attribute.attencoding_context`, backend cache, context
> group
> lazy fallback, `ResolveEncodingContextOid`, `pg_depend`/`ObjectAddress`
> integration, SQL management/inspection functions, built-in handlers,
> and
> regression tests (`src/test/regress/sql/encoding_context.sql`).
> - Patch 0002 (follow-up): Native SQL DDL grammar (`CREATE/ALTER/DROP
> ENCODING
> TYPE`, `CREATE/ALTER/DROP ENCODING CONTEXT`, `ALTER TABLE ... ALTER
> COLUMN
> ... SET/DROP ENCODING CONTEXT`) and `COMMENT ON` / `SECURITY LABEL` /
> `ALTER ... RENAME` / `SET SCHEMA` / `OWNER TO` support.
> - Patch 0003 (follow-up, building on Direct TOAST): Per-chunk and
> whole-datum
> transparent TOAST encoding/decoding via `chunk_ectx_id`, O(log N) chunk
> slice decompression, and zero-rewrite online context switching.
>
> Feedback on the catalog design, Context Group (`ectxgroup` / `ectxvariant`)
> fallback semantics, and C/SQL APIs is very welcome.
>
> Best regards,
> Hannu Krosing
>