Auth extensions, with an LDAP/SCRAM example [was: Proposal: Support custom authentication methods using hooks]

(Splitting off a new thread, and mostly clearing the CC list so people
can escape if they want, because this is a heckuva tangent.)

TL;DR: the attached patchset lets you authenticate with Postgres by
forwarding SCRAM authentication to a backend LDAP server, and I use it
to make arguments for long-term plans.

= Motivation =

Upthread (and in other threads) there have been a bunch of related open
questions around authentication:

- Should our authn/z framework be extensible, or should core provide the
only supported implementations?
- If it's extensible, are server-side extensions useful by themselves,
or do we need to have client-side extension points as well?
- What would a good extension API look like? Should extensions be able
to switch out core's implementation of the exchange (e.g. entire SASL
mechanisms) or should they just provide individual authentication
details to core (e.g. SCRAM secrets)?
- Plaintext auth is terrible. Can we get rid of it?

Additionally, there have been recent conversations about the next steps
for our SCRAM implementation:

- Do we need to maintain RFC compatibility, or is it sufficient for
libpq and Postgres to speak the same non-standard flavor?
- Do we want to maintain multiple channel binding types? Can we make use
of endpoint bindings effectively or should we switch to unique?

I like talking about code I can see, so I've written an experimental
feature that touches on several of these points at once.

= An LDAP/SCRAM Bridge =

So this thing, based on Samay's server-side auth hooks, authenticates
the user by forwarding the client's SCRAM header to an LDAP server. If
the LDAP exchange succeeds, the user is allowed in. (The security of
that hinges on ensuring that the Postgres user and the SCRAM/LDAP user
are in fact the same.)

I see this architecture -- if not necessarily the implementation -- as a
straight upgrade from plaintext LDAP auth. The secret remains embedded
in LDAP, never exposed. So the Postgres server can't log in as the user
again after it drops its backend connection, and it can't even pretend
to be the LDAP server in the future, though it can still use that
in-progress connection attempt to act on behalf of the user.

(I hear you asking, "doesn't this break with channel bindings, since
they prevent MITM attacks?" and the answer is "no, or at least it's not
supposed to." We implement an endpoint binding -- which permits MITMs
*if* they also hold the server's private key -- instead of a unique
binding. In other words, we explicitly allow proxied authentication
between servers holding the same certificate.

Unfortunately, OpenLDAP 2.6 implements something it *calls*
tls-server-end-point but is *not*, so if you want to see that in action
you have to patch OpenLDAP. Which is not very helpful for my argument.
Ah well.)

This patchset should ideally have required zero client side changes, but
because our SCRAM implementation is slightly nonstandard too -- it
doesn't embed the username into the SCRAM data -- libpq can't talk to
the OpenLDAP/Cyrus SASL implementation. I included a quick-and-bad patch
to fix it in libpq; that needs its own conversation.

= My Arguments =

Okay, so going back to the open questions:

I think this is exactly the sort of thing that's too niche to be in core
but might be extremely useful for the few people it applies to, so I'm
proposing it as an argument in favor of general authn/z extensibility.

Furthermore, this is an example of a server-side-only extension that is
useful by itself and is still compatible with a (SCRAM-compliant) client.

This approach works best by switching out the entire implementation of
the SCRAM SASL mechanism. You wouldn't be able to implement this
architecture with a more targeted, secrets-focused API. (And personally
I think the idea of exchanging SCRAM secrets between machines is
malpractice. Those are the building blocks of user trust; we're supposed
to protect them accordingly.)

Keeping the OAuth thread in mind, I still think it would be most helpful
to develop and switch out SASL mechanisms on *both* the server and
client side, independently of the Postgres major version. (Which means
letting third parties do it too. I know there are valid concerns about
that; I still want Postgres to have the best general implementations.)

My preference would be to use an existing pluggable SASL implementation
rather than growing our own. But when we chose not to implement SCRAM to
the letter, we tied our own hands a bit. I briefly tried porting the
server side to two separate third-party SASL libraries, and I
immediately ran into compatibility issues with the SCRAM username (not
included in the libpq flavor of SCRAM, as noted above) and the default
channel binding (endpoint instead of unique). We'd probably need to
backport compatibility fixes if we wanted to roll out third-party SASL.

Finally, I think allowing a server to proxy your authority via an
endpoint binding is something you should be able to opt into -- or at
least opt out of? And I don't want to remove endpoint bindings entirely,
because I think this functionality is potentially very useful. I'd like
to see us support both, and eventually default to unique.

= Patch Roadmap =

- 0001-4 are Samay's server-side auth hook set, rebased over HEAD.
- 0005 is a quick client-side hack to include the username in the SCRAM
header, making it compliant enough to interoperate in the simplest
cases. (It's not compliant enough to be safe, yet.)
- 0006 is the actual implementation, including tests so you can try it
out too.
- Finally, the OpenLDAP-* patch is for OpenLDAP 2.6.4; it's a hack to
implement standard channel bindings so you can prove to yourself that
they work. If you apply this, set $ldap_supports_channel_binding in the
Perl test so you can see it working.

Thanks,
--Jacob