Re: Channel binding for post-quantum cryptography

On Sun, Oct 26, 2025 at 11:20:53AM +0100, Filip Janus wrote:
> I have prepared a test case following the pattern from commit 9244c11afe23
> (RSA-PSS fix).

Thanks, I'm able to reproduce your problem with the error you have,
after generating the certs.

+ my $mldsa_cert = "ssl/server-mldsa65.crt";
+ skip "ML-DSA-65 requires OpenSSL 3.5+ for certificate generation",1
+ unless -f $mldsa_cert;

The certs are stored in the tree, regenerated by a `make sslfiles` run
in src/test/ssl/. We cannot rely on such a check to decide if this
scenario should be skipped or not. In past branches, like
REL_13_STABLE, one example of a "correct" way is done in 002_scram.pl
with HAVE_X509_GET_SIGNATURE_NID, where we rely on a compile check
when running the test.

> You are correct that, according to RFC 5929, we should ideally use the hash
> function associated with the certificate's signatureAlgorithm. However, if
> I understand it correctly, there are distinctions with ML-DSA:
> I investigated OpenSSL's API to retrieve the hash algorithm used by ML-DSA,
> and I haven't found a suitable solution.
>
> ML-DSA seems to have an internal structure but no public API to extract
> SHAKE128/256 configuration.

Hmm. Has this question been asked to upstream OpenSSL? Perhaps their
reply would be "you-are-doing-it-wrong", but it may be something where
their input may drive the implementation.

> The ML-DSA Specifies
>
> ML-DSA (FIPS 204) uses SHAKE internally:
> - ML-DSA-44: SHAKE128 (128-bit security)
> - ML-DSA-65: SHAKE256 (192-bit security)
> - ML-DSA-87: SHAKE256 (256-bit security)

Yeah, I've been reading around this area as well, while browsing the
code:
https://github.com/openssl/openssl/blob/master/doc/designs/ml-dsa.md
https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.204.pdf

There are traces in the OpenSSL code of the following things, not sure
if these could point at something:
NID_HASH_ML_DSA_44_WITH_SHA512
NID_HASH_ML_DSA_65_WITH_SHA512
NID_HASH_ML_DSA_87_WITH_SHA512

> ML-DSA doesn't have an "associated" hash function in the sense that
> RSA-SHA256 does. The SHAKE function is internal to the signing process, not
> a separate digest step. For the purpose of channel binding (hashing the
> entire certificate), we need a traditional hash function. So that's why
> I've chosen SHA-256

Another thing that bugs me is that this patch would force sha-256 for
everything, without at least checks based on NID_ML_DSA_44,
NID_ML_DSA_65 or NID_ML_DSA_87. That may be more flexible, but I'm
wondering if it could become a problem long-term to enforce blindly
such a policy every time algo_nid is undefined.

> Regarding NIDs and Future Extensions, I would expect growth, but I am not a
> security specialist.

This needs more study on my part, at least. Adding a couple more
folks in CC for now. Perhaps they have an opinion on the matter, I am
not the most familiar with these new toys in OpenSSL 3.5.

Anyway, even with these points, I am not much a fan of adding again
a dependency to X509_get_signature_nid() while we have switched to
X509_get_signature_info() to be able to support RSA-PSS signatures.
It is annoying to have to rely again on X509_get_signature_nid() for
what's a new special case, NID_undef being the synonym of an error
usually, and that's what EVP_get_digestbynid() is complaining about in
this case.
--
Michael