Re: Channel binding for post-quantum cryptography

Hi,

Thank you for the detailed feedback. Let me address your points:

Test Case
=========

I have prepared a test case following the pattern from commit 9244c11afe23
(RSA-PSS fix).

Regarding the Hash Algorithm
=============================

You are correct that, according to RFC 5929, we should ideally use the hash
function associated with the certificate's signatureAlgorithm. However, if
I understand it correctly, there are distinctions with ML-DSA:
I investigated OpenSSL's API to retrieve the hash algorithm used by ML-DSA,
and I haven't found a suitable solution.

ML-DSA seems to have an internal structure but no public API to extract
SHAKE128/256 configuration

The ML-DSA Specifies

ML-DSA (FIPS 204) uses SHAKE internally:
- ML-DSA-44: SHAKE128 (128-bit security)
- ML-DSA-65: SHAKE256 (192-bit security)
- ML-DSA-87: SHAKE256 (256-bit security)

However, this is a different approach from traditional signature algorithms:
- Traditional (e.g., RSA-SHA256): Hash is a separate, associated algorithm
that can be queried
- ML-DSA: SHAKE is an internal component of the signature algorithm, not an
associated digest for external use

ML-DSA doesn't have an "associated" hash function in the sense that
RSA-SHA256 does. The SHAKE function is internal to the signing process, not
a separate digest step. For the purpose of channel binding (hashing the
entire certificate), we need a traditional hash function. So that's why
I've chosen SHA-256

SHA-256 is appropriate because:
1. It matches the security level of ML-DSA-65 (both ~256-bit security)
2. RFC 5929 recommends SHA-256 for unknown/unsupported algorithms
3. It's the standard fallback used throughout the codebase
4. SHAKE256 (via EVP_shake256()) is an XOF (eXtendable Output Function),
not a traditional fixed-size hash suitable for this use case

Regarding NIDs and Future Extensions, I would expect growth, but I am not a
security specialist.

Current Patch Status
====================

I'm keeping the current patch as-is for now, and I am adding the requested
test case in a separate commit.

-Filip-

po 20. 10. 2025 v 10:06 odesílatel Michael Paquier <michael(at)paquier(dot)xyz>
napsal:

> On Mon, Oct 20, 2025 at 09:12:52AM +0200, Filip Janus wrote:
> > The problem is caused by a difference between the currently used
> algorithms
> > and post-quantum ones. For example, commonly used algorithms like RSA
> have
> > a defined digest algorithm, but ML-DSA does not.
> >
> > PostgreSQL's channel binding implementation expects all signature
> > algorithms to have a traditional digest mapping, but post-quantum
> > algorithms such as ML-DSA use their hash function internally as part of
> the
> > signature process.
>
> Noted.
>
> > As a result, the connection fails with the following error:
> >
> > could not find digest for NID UNDEF
> >
> > The issue can be worked around by disabling channel binding.
> >
> > Although the RFC is not entirely clear on how to handle this situation,
> in
> > my patch I propose using SHA-256 as the default digest in such cases.
>
> Based on the RFC at [1], we have indeed:
> if the certificate's signatureAlgorithm uses a single hash function
> and that hash function neither MD5 nor SHA-1, then use the hash
> function associated with the certificate's signatureAlgorithm;
>
> So it would be essential to reuse the hash function used by this
> algorithm. Except that you are saying that we have no way to retrieve
> that, even if it's a different routine than EVP_get_digestbynid()?
> Enforcing blindly SHA-256 does not seem to be the right move because
> it could enforce an incorrect behavior, even if it would be more
> user-friendly in some cases like this one.
>
> So, X509_get_signature_info() is able to return an algo NID that we
> can then use to decide which algo type we should take. I can see
> three NIDs associated to ML-DSA: NID_ML_DSA_44, NID_ML_DSA_65 and
> NID_ML_DSA_87. Could this list grow?
>
> Based on ml-dsa.md, I am wondering if we could do something based on
> ML_DSA_PARAM. I am not sure, but OpenSSL, while being a spaghetti
> code base, usually has some internal way to extract some of its
> contents. Saying that, they tend to hide more internals behind
> pointers, 3.0 has added some of that stuff.
>
> A good first step would be to design a reproducible test case to
> investigate the issue. Could it be possible to craft a test case that
> could then be added into the tree? We have automated tests in
> src/test/ssl/. See for example the level of craft done for a similar
> past issue, as of commit 9244c11afe23. That would be the minimum
> required for a potential fix.
>
> Note that the use of X509_get_signature_info() is conditional in our
> code, so your patch would likely fail to compile depending on the
> version of OpenSSL linked to. The minimum version of OpenSSL
> supported by PG on HEAD is 1.1.1. On the oldest stable branches (v13
> or v14), this requirement is.. Cough.. 1.0.1.
>
> [1]: https://datatracker.ietf.org/doc/html/rfc5929#section-4.1
> --
> Michael
>