Re: Proposal: new file format for hba/ident/hosts configuration?

Hello

I reworked the prototype based on the feedback in this thread, and
attached a new patchset implementing it.
The patches are marked as POC on purpose: they are complete enough to
try out (docs, tests, initdb support), but they are definitely not
ready for proper code review yet.

I'm still mainly looking for feedback on the idea and the format,
before spending more time on a proper implementation.

The main changes since my last emails:

1. It is JSON now, not TOML.

Most answers preferred json(5), mainly because we already have a
parser for it. So this version uses the existing src/common JSON
parser, without any new dependency.

The patchset only accepts strict JSON for now. I sent my JSON5 patches
to a separate thread[1], and if that gets accepted, accepting JSON5
here (comments, trailing commas, unquoted keys, ...) is a parser flag.

The current examples are noisier than they would be with JSON5, and
there is no way to write comments, which is a real limitation for a
configuration file.

The JSON5 patchset is ready for review, so if somebody has time and is
interested in it, please check it out.

2. hba is a tree, not a flat array of rules.

Jacob found the array of rules hard to read, and asked for "contexts
with property bags", like web servers do. So instead of a 1:1 mapping
of pg_hba.conf with _defaults and based_on, an hba node now narrows
the match of its parent, and carries authentication options and
settings that its children inherit (integrating my previous thread
about custom OAuth validator options[2]).

Children are tried before their parent, so the most specific node
wins, and a flat pg_hba.conf is simply a tree of depth one.

Jacob's example from upthread looks like this:

"hba": {"children": [
{"auth": {"method": "ldap", "ldapserver": "ldap.example.com", "ldaptls": 1},
"children": [
{"users": "+admins", "auth": {"clientcert": "verify-full"}},
{"users": "jacob", "auth": {"method": "reject"}}
]}
]}

Greg asked how the default pg_hba.conf would look, that is:

"hba": {"children": [
{"transport": "local", "auth": {"method": "trust"}},
{"address": ["127.0.0.1/32", "::1/128"], "auth": {"method": "trust"}},
{"databases": "replication", "children": [
{"transport": "local", "auth": {"method": "trust"}},
{"address": ["127.0.0.1/32", "::1/128"], "auth": {"method": "trust"}}
]}
]}

Narrowing is a convention, not enforced: a child can name users or
addresses outside of its parent's. I'm not sure if this is acceptable,
or if this makes the tree misleading.

3. Per-rule and per-host settings.

This is the "this group of users should use the following GUCs" part,
connected to the hba GUC discussion[2]. hba nodes and hosts entries
can have a "settings" object, applied when they match. For this the
patch adds 2 new GUC contexts, PGC_HOSTS and PGC_HBA, between SIGHUP
and SU_BACKEND. Currently only authentication_timeout is moved into
PGC_HBA, so it can differ per authentication method. Everything later
than PGC_HBA (e.g. work_mem, statement_timeout) can be set per rule
already.

4. The "final state", including postgresql.conf.

Jacob asked for a mockup of the whole thing, so this is a "complete"
patchset, and the last patches add a postgresql.json root file: GUCs
at the top level, and "hba", "ident" and "hosts" sections in the same
document, with "include" for splitting it into multiple files. initdb
--config-format=json creates a cluster with it.

I also attached a larger example with all of the above
(example-postgresql.json). With the patchset applied and the paths
adjusted, it works as a complete configuration.

Everything is still completely opt-in in the PoC patch, legacy
configurations continue working the same way.

Things I left out, and know about:

* A conversion tool. Andrew and Daniel both said that we need one. I
considered adding one to this PoC patchset, but decided to leave it
out for now.
* JSON strings are used as bytes, as in postgresql.conf, so non-ASCII
values have the same encoding issues as today.

Main open questions:

* What is the exact scope? The attached patchset is a mostly complete
PoC to show the vision, but as I wrote earlier in this thread, I am
not sure that all this should be done in one step.
* What does everyone think about this format proposal, compared to the
versions I sent earlier?
* Does anybody have different ideas/suggestions?

[1]: https://www.postgresql.org/message-id/CAN4CZFMmapQG7yJZwjxKhZ%3DzjzDrEK2GYCg%3DFo2_47UYCrkUPA%40mail.gmail.com
[2]: https://www.postgresql.org/message-id/CAN4CZFM3b8u5uNNNsY6XCya257u%2BDofms3su9f11iMCxvCacag%40mail.gmail.com