[PATCH v1] Generic Datum Encoding and Compression Contexts (pg_encoding_type, pg_encoding_context)

Hi hackers,

Attached is a patch introducing a generic, extensible Datum Encoding and
Compression Context subsystem for PostgreSQL (`pg_encoding_type` and
`pg_encoding_context`).

1. Motivation
=============

Currently, column-level compression in PostgreSQL is controlled by the single
`char` column `pg_attribute.attcompression` ('p' for pglz, 'l' for lz4) and
operates solely as a stateless, parameterless byte-stream transformation.

In practice, many high-value storage and analytical workloads benefit from
stateful, pre-trained, or schema-aware encoding metadata stored in system
catalogs and cached in backend memory:

- Dictionary-based compression (e.g., Zstd pre-trained dictionaries for JSONB
or log payloads, FSST symbol tables for short strings) where per-datum
compression ratios improve dramatically when a shared dictionary is
factored out of individual tuples.
- Vector quantization and entropy coding (e.g., scalar quantization min/max
calibration bounds, tANS / asymmetric numeral systems tables, or custom
codebooks) where a column or type modifier references a trained context.
- Schema-aware binary serialization (e.g., Avro or Protobuf schemas) and
multi-stage pipelines (e.g., schema encoding followed by dictionary
compression).
- Online dictionary / schema evolution without a full table rewrite: when a
better dictionary or new schema version is trained, new writes should use
the new context while historical tuples remain self-describing and readable
using the context with which they were written.
- Multi-variant fallback within a single column ("Context Groups"): a column
may prefer a specialized encoding variant (e.g., a restricted hex/UUID
alphabet or a strict schema) that falls back automatically to a more
general variant if a particular input datum cannot be represented by the
stricter variant, using only a 1-byte (`uint8`) variant selector in the
encoded payload.

2. Overview of the Design
=========================

2.1 System Catalogs
-------------------

* `pg_encoding_type` (OID 9440):
Registers pluggable encoding engines (built-in or extension-provided):
- `oid` (`oid`)
- `enctypname` (`name`), `enctypnamespace` (`oid`), `enctypowner` (`oid`)
- `enctypkind` (`char`):
'd' = dictionary, 'c' = compression, 'q' = quantization, 'e' = custom
- `enctyptype` (`text`): engine identifier (e.g., "pglz",
"zstd", "xor_lut")
- `enctypencfunc` (`regproc`): `enc_fn(internal, anyelement) -> bytea`
- `enctypdecfunc` (`regproc`): `dec_fn(internal, bytea) -> anyelement`
- `enctypexpfunc` (`regproc`, optional): `exp_fn(oid, text) -> json`
- `enctypoptions` (`text[]`): default key=value options

* `pg_encoding_context` (OID 9431):
Stores trained or configured encoding context instances:
- `oid` (`oid`)
- `ectxname` (`name`), `ectxnamespace` (`oid`), `ectxowner` (`oid`)
- `ectxtypid` (`oid` -> `pg_encoding_type.oid`)
- `ectxtargettypid` (`oid` -> `pg_type.oid`, or 0 for polymorphic/any)
- `ectxwriteok` (`bool`, default `true`): whether this context may be
used for encoding new data; setting `ectxwriteok = false` retires a
context for new writes while keeping it active for decoding existing
tuples.
- `ectxgroup` (`oid` -> `pg_encoding_context.oid`): context group OID
stored in `pg_attribute.attencoding_context`. Always equals `oid` for
standalone contexts and group leaders, and equals the group leader's
`oid` for additional variants in the group.
- `ectxvariant` (`uint8`, `0..255`, default `0`): 1-byte variant selector
within `ectxgroup`. `0` designates a standalone context
(`ectxgroup = oid`,
variants not supported). Non-zero values (`1..255`) identify and order
variants within a group.
- `ectxlabel` (`text`): human-readable version/schema label
- `ectxenctab` (`bytea`), `ectxdectab` (`bytea`): raw encoding and
decoding dictionaries/tables (TOAST-able)
- `ectxoptions` (`text[]`): per-context key=value options

* `pg_attribute.attencoding_context` (`oid`, default `0`):
Binds a column to an encoding context (or context group leader) for new
writes, tracked via `pg_depend` (`DEPENDENCY_NORMAL`).

2.2 Context Groups (`ectxgroup` / `ectxvariant`) and Lazy Fallback
------------------------------------------------------------------

When a column can be encoded by multiple related contexts (for example, a
high-density restricted-alphabet encoder as primary variant `1` and a general
compressor as fallback variant `2`), storing a 4-byte OID per inline value is
unnecessary overhead when a 1-byte (`uint8`) variant indicator suffices.

- `pg_attribute.attencoding_context` stores the 4-byte `ectxgroup` OID.
- Each member of a group has a unique `ectxvariant` in `1..255`.
- When encoding with a group context, `EncodingContextEncodeWithEffectiveOid`
starts at the lowest `ectxvariant` in the group. If a variant has
`ectxwriteok = false` or its encode handler declines/fails to encode the
input datum, execution automatically falls back to the context with the
same `ectxgroup` and the next higher `ectxvariant`, returning both the
encoded `bytea` and the effective context OID (`*used_context_id`) that
succeeded.
- In memory, `EncodingContextData.fallback` is initialized to
`ECTX_UNRESOLVED` (`(EncodingContextData *)(uintptr_t) 1`) when
`variant > 0` (and `NULL` when `variant == 0`), and is lazily resolved on
first fallback via `EncodingContextGetFallback()` by scanning the group for
the smallest `ectxvariant > current->variant`.

2.3 Backend Cache & Invalidation
--------------------------------

`LookupEncodingContext(Oid)` and
`LookupEncodingContextByGroupVariant(Oid, uint8)`
materialize `EncodingContextData` entries into a session-local `dynahash` table
allocated inside `EncodingContextMemoryContext` (a child of
`CacheMemoryContext`).

- Each cached entry has its own child `MemoryContext` so individual entries
can be freed cleanly without leaking or resetting sibling contexts.
- `CacheRegisterSyscacheCallback` hooks on `ENCODINGCONTEXTOID` and
`ENCODINGTYPEOID` invalidate cached entries on `ALTER` or `DROP`, and reset
resolved `fallback` pointers back to `ECTX_UNRESOLVED` so group membership
changes take effect immediately across backends.
- Active encode/decode calls pin entries via `ectx->refcount`; if an entry is
invalidated while pinned on the call stack, destruction is deferred until
the outermost caller releases its pin.

2.4 Common Name / Numeric OID Resolution
----------------------------------------

To provide uniform context resolution across core SQL functions, DDL, and
extension `typmodin` handlers (such as `sqvector(dim, 'ctx_name')` or
`sqvector(dim, 16385)`):

- C API: `Oid ResolveEncodingContextOid(const char *name_or_oid,
bool missing_ok)`
- SQL function: `pg_resolve_encoding_context_oid(name_or_oid text,
missing_ok bool DEFAULT false) RETURNS oid`

If the input string consists of decimal digits corresponding to an existing
`pg_encoding_context.oid`, that OID is returned directly; otherwise the string
is parsed as an identifier (supporting schema-qualified `schema.name` or
`search_path` resolution).

2.5 SQL Functions & Built-in Reference Handlers
-----------------------------------------------

This patch provides SQL-callable functions for managing, testing, and
inspecting encoding types and contexts:

- `pg_create_encoding_type`, `pg_drop_encoding_type`
- `pg_create_encoding_context`, `pg_alter_encoding_context`,
`pg_drop_encoding_context`
- `pg_set_column_encoding_context`
- `pg_resolve_encoding_context_oid`
- `pg_explain_encoding_context(context_id oid, direction text
DEFAULT 'encoding') -> json`
- `pg_encode_datum(context_id oid, val anyelement) -> bytea`
- `pg_decode_datum(context_id oid, payload bytea, target_type
anyelement) -> anyelement`
- `pg_column_raw_encoding(val anyelement) -> bytea`

And two built-in reference encoding engines:
- `pg_bytea_dict_encode` / `pg_bytea_dict_decode` / `pg_bytea_dict_explain`
(supporting both XOR dictionary transformations and restricted-alphabet
lookup tables for testing multi-variant group fallback)
- `pg_pglz_context_encode` / `pg_pglz_context_decode` /
`pg_pglz_context_explain`

3. Patch Roadmap
================

To keep review manageable, this work is structured as a 3-part series:

- Patch 0001 (attached): Core `pg_encoding_type` and `pg_encoding_context`
catalogs, `pg_attribute.attencoding_context`, backend cache, context group
lazy fallback, `ResolveEncodingContextOid`, `pg_depend`/`ObjectAddress`
integration, SQL management/inspection functions, built-in handlers, and
regression tests (`src/test/regress/sql/encoding_context.sql`).
- Patch 0002 (follow-up): Native SQL DDL grammar (`CREATE/ALTER/DROP ENCODING
TYPE`, `CREATE/ALTER/DROP ENCODING CONTEXT`, `ALTER TABLE ... ALTER COLUMN
... SET/DROP ENCODING CONTEXT`) and `COMMENT ON` / `SECURITY LABEL` /
`ALTER ... RENAME` / `SET SCHEMA` / `OWNER TO` support.
- Patch 0003 (follow-up, building on Direct TOAST): Per-chunk and whole-datum
transparent TOAST encoding/decoding via `chunk_ectx_id`, O(log N) chunk
slice decompression, and zero-rewrite online context switching.

Feedback on the catalog design, Context Group (`ectxgroup` / `ectxvariant`)
fallback semantics, and C/SQL APIs is very welcome.

Best regards,
Hannu Krosing