OpenSSL compat with ML-DSA by kojo1 · Pull Request #10962 · wolfSSL/wolfssl

kojo1 · 2026-07-21T21:35:01Z

Description

wolfSSL_PEM_read_bio_PrivateKey
wolfSSL_PEM_read_PrivateKey
wolfSSL_d2i_PrivateKey, wolfSSL_d2i_PublicKey
wolfSSL_X509_sign
wolfSSL_X509_REQ_sign
wolfSSL_X509_set_pubkey

Fixes zd#22151

Testing

Added test_wolfSSL_PEM_PrivateKey_mldsa
Added ML-DSA cases in test_wolfSSL_X509_set_pubkey

Checklist

added tests
updated/added doxygen
updated appropriate READMEs
Updated manual and documentation

github-actions · 2026-07-21T21:49:49Z

MemBrowse Memory Report

gcc-arm-cortex-m4

FLASH: .text +64 B (+0.0%, 200,917 B / 262,144 B, total: 77% used)

gcc-arm-cortex-m4-openssl-compat

FLASH: .text +640 B (+0.1%, 771,668 B / 1,048,576 B, total: 74% used)

gcc-arm-cortex-m4-pkcs7

FLASH: .text +64 B (+0.0%, 213,908 B / 262,144 B, total: 82% used)

gcc-arm-cortex-m4-rsa-only

FLASH: .text -64 B (-0.0%, 326,104 B / 1,048,576 B, total: 31% used)

gcc-arm-cortex-m4-tls13

FLASH: .text -64 B (-0.0%, 236,615 B / 262,144 B, total: 90% used)

gcc-arm-cortex-m7-pq

FLASH: .text -64 B (-0.0%, 280,512 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m7-tls13

FLASH: .text -64 B (-0.0%, 236,679 B / 262,144 B, total: 90% used)

linuxkm-pie

Data: __patchable_function_entries +88 B (+0.3%, 26,344 B)

linuxkm-standard

Data: __patchable_function_entries +64 B (+0.1%, 49,040 B)

stm32-sim-stm32h753

FLASH: .text +1,792 B (+1.0%, 185,684 B / 2,097,152 B, total: 9% used)
No memory changes detected for:
gcc-arm-cortex-m0plus
gcc-arm-cortex-m3
gcc-arm-cortex-m4-baremetal
gcc-arm-cortex-m4-crypto-only
gcc-arm-cortex-m4-dtls13
gcc-arm-cortex-m4-min-ecc
gcc-arm-cortex-m4-pq
gcc-arm-cortex-m4-sp-math
gcc-arm-cortex-m4-tls12
gcc-arm-cortex-m7

Frauschi

🐺 Skoll Code Review

Overall recommendation: APPROVE
Findings: 4 total — 3 posted, 2 skipped

Posted findings

[High] ML-DSA private key decode in X509 sign/set_pubkey missing PRIVATE_KEY_UNLOCK bracket (fails under FIPS) — src/x509.c:12827, src/x509.c:16508
[Medium] WC_MAX_X509_GEN increase puts 20 KB on the stack in wolfSSL_X509_REQ_sign for all key types — src/x509.c:12953-12962, 16922-16923
[Medium] wolfSSL_X509_REQ_sign changes lack ML-DSA test coverage — src/x509.c:16918-16950

Skipped findings

[Low] fnames[] can become a zero-length array if all ML-DSA levels are disabled
[Medium] WC_MAX_X509_GEN increase puts 20 KB on the stack in wolfSSL_X509_REQ_sign for all key types

Review generated by Skoll via Claude/Codex

kojo1 · 2026-07-24T21:51:32Z

lack ML-DSA test coverage

Add ML-DSA test coverage for X509_REQ_sign

missing PRIVATE_KEY_UNLOCK bracket

PRIVATE_KEY_UNLOCK to X509 sign/set_pubkey

WC_MAX_X509_GEN increase puts 20 KB on the stack

Dynamic buff size based on expected cert size

Frauschi · 2026-07-27T07:56:01Z

@kojo1 The CAVP self test failure is legit (undefined references during linking).

Frauschi

🐺 Skoll Code Review

Overall recommendation: APPROVE
Findings: 19 total — 13 posted, 6 skipped

Posted findings

[High] d2iTryMlDsaKey silently returns a stale EVP_PKEY when the caller reuses an existing key object — wolfcrypt/src/evp_pk.c:1086-1096
[Medium] DER buffer sizing ignores the subject public key, only the signing key — src/x509.c:12961-12974
[Medium] ML-DSA signing still requires a valid digest; comment claims md is ignored — src/x509.c:12274-12277
[Medium] Preprocessor gating mismatch: sigTypeFromPKEY accepts ML-DSA in builds where resign_cert cannot sign — src/x509.c:12274
[Medium] MlDsaKey reused across a failed PrivateKeyDecode without Free/Init in X509_set_pubkey — src/x509.c:16519-16542
[Medium] d2i_evp_pkey PKCS#8 algorithm cross-check not extended for ML-DSA — wolfcrypt/src/evp_pk.c:1548-1553
[Medium] Test coverage limited to ML-DSA-44; the riskiest new paths are untested — tests/api/test_x509.c:1071-1128
[Medium] wolfSSL_X509_set_pubkey stores the standard ML-DSA OID for FIPS204-draft keys; the WC_ML_DSA_*_DRAFT cases are unreachable — src/x509.c:16546-16590
[Low] New ML-DSA code uses MlDsaKey and mixed heap hints, unlike the rest of x509.c — src/x509.c:12786
[Low] WC_MAX_X509_GEN_MLDSA is an unrelated magic number to WC_MAX_X509_GEN — src/x509.c:12962-12974
[Low] X509_GEN_BUF_SZ sizes the DER buffer from the signing key, ignoring an ML-DSA subject public key — src/x509.c:12961-12974
[Low] New ML-DSA code uses legacy-only LEVEL2/3/5 spellings, breaking WOLFSSL_NO_DILITHIUM_LEGACY_NAMES builds — src/x509.c:12279-12295
[Info] ML-DSA key object reused across a failed private-key decode and a public-key decode without re-init — src/x509.c:16527-16545

Skipped findings

[Medium] WC_DECLARE_VAR/WC_ALLOC_VAR_EX abstraction dropped in X509_REQ_sign
[Low] ML-DSA level-to-OID mapping duplicated three times; four-macro guard repeated four times
[Low] Test guards: zero-length array risk and missing NO_SHA256 condition
[Info] New test entry breaks the backslash alignment of TEST_X509_DECLS
[Info] ML-DSA private key DER newly persisted in EVP_PKEY.pkey.ptr, which is freed without ForceZero
[Info] X509_GEN_BUF_SZ evaluates its argument twice and is never #undef'd

Review generated by Skoll via Claude/Codex

Frauschi

🐺 Skoll Code Review

Overall recommendation: REQUEST_CHANGES
Findings: 9 total — 8 posted, 1 skipped

Posted findings

[Medium] X509_set_pubkey() fails for ML-DSA private keys that carry no seed and no embedded public key — src/x509.c:16598
[Medium] Uninitialized MlDsaKey / WC_RNG are freed unconditionally when an earlier Expect short-circuits — tests/api/test_ossl_x509_pk.c:488
[Medium] Double XFCLOSE on the same XFILE: the f = XBADFILE reset is missing after the second close — tests/api/test_evp_pkey.c:2826
[Medium] New tests still use the legacy-only MlDsaKey typedef, breaking WOLFSSL_NO_DILITHIUM_LEGACY_NAMES builds — tests/api/test_evp_pkey.c:2846
[Medium] EVP_PKEY reuse still leaks the previously attached algorithm object, and the new test hides it — wolfcrypt/src/evp_pk.c:68
[Low] sigOID is left set to WOLFSSL_FAILURE on the X509/REQ object when the signature-type lookup fails — src/x509.c:13029
[Low] Private-key DER is released without zeroization when an EVP_PKEY is reused — wolfcrypt/src/evp_pk.c:71
[Low] A non-NULL digest is silently ignored for ML-DSA signing — src/x509.c:12206

Skipped findings

[Low] The new reuse block indexes fnames[] unconditionally, which can be a zero-length array

Review generated by Skoll via Claude/Codex

Frauschi

🐺 Skoll Code Review

Overall recommendation: COMMENT
Findings: 12 total — 3 posted, 9 skipped

Posted findings

[Medium] d2i_free_prev_key_obj() does not mirror EVP_PKEY_free() for X25519/X448/HKDF/CMAC, so those objects leak on reuse — wolfcrypt/src/evp_pk.c:44-107
[Low] Ed25519/Ed448 d2i reuse leaves the stale type and DER in place - pre-existing, but the prePopulated pattern this PR adds is the fix — wolfcrypt/src/evp_pk.c:1180
[Medium] New tests add several multi-KB stack buffers — tests/api/test_ossl_x509_pk.c:441,470,501,545; tests/api/test_evp_pkey.c:2735-2737,2790-2792

Skipped findings

[Low] The new ForceZero assumes pkey_sz describes the pkey.ptr allocation - true today, but the invariant is already broken on master (pre-existing, not a blocker)
[Medium] Unrelated deletion of WOLFSSL_MEM_FAIL_COUNT from .wolfssl_known_macro_extras
[Info] FALSE POSITIVE - X509_GEN_BUF_SZ / WOLFSSL_MLDSA_X509_SIGN definition block cannot be skipped while a consumer is compiled
[Medium] No test for the 'sigOID unchanged on rejected key/md' behavior the PR adds
[Info] ALREADY RESOLVED - non-NULL md silently ignored for ML-DSA
[Low] X509_GEN_BUF_SZ evaluates its pkey argument twice and drops it entirely in the non-ML-DSA variant
[Low] Reused EVP_PKEY is left gutted when the d2i copy allocation fails
[Low] Conditional free of pkey inside the key-file loop is hard to follow
[Info] Continuation backslash misaligned in TEST_X509_DECLS

Review generated by Skoll via Claude/Codex

Frauschi · 2026-08-03T09:45:12Z

+ *
+ * @param [in, out] pkey  EVP PKEY being repurposed.
+ */
+static void d2i_free_prev_key_obj(WOLFSSL_EVP_PKEY* pkey)


🟡 [Medium] d2i_free_prev_key_obj() does not mirror EVP_PKEY_free() for X25519/X448/HKDF/CMAC, so those objects leak on reuse
💡 SUGGEST bug

The new helper's comment says it mirrors "the per-type cleanup in wolfSSL_EVP_PKEY_free()", but it only covers RSA, EC, DSA, DH, ED25519 and ED448. wolfSSL_EVP_PKEY_free() (wolfcrypt/src/evp.c:12332-12368) additionally handles WC_EVP_PKEY_X25519 (curve25519), WC_EVP_PKEY_X448 (curve448), WC_EVP_PKEY_HKDF (hkdfSalt/hkdfKey/hkdfInfo) and WC_EVP_PKEY_CMAC (cmacCtx).

Because d2i_make_pkey() overwrites pkey->type right after calling the helper, an object of one of those four types is not merely left attached - it becomes permanently unreachable, since wolfSSL_EVP_PKEY_free() switches on the new type and never visits the old case. That is the same defect this PR just fixed for RSA, only for the types the helper missed.

Confirmed with a PoC against the PR head (--enable-all --enable-dilithium --enable-experimental, macOS leaks):

pkey = wolfSSL_EVP_PKEY_new_raw_public_key(WC_EVP_PKEY_X25519, NULL, raw, 32); p = client_keypub_der_2048; wolfSSL_d2i_PUBKEY(&pkey, &p, sizeof_client_keypub_der_2048); /* reuse */ wolfSSL_EVP_PKEY_free(pkey);

Process 42402: 1 leak for 160 total leaked bytes. STACK OF 1 INSTANCE OF 'ROOT LEAK: <malloc in wolfSSL_EVP_PKEY_new_raw_public_key>'

The same run confirms the RSA case this PR fixed is now clean: an RSA-then-RSA d2i_PUBKEY reuse leaks nothing and the pkey->rsa pointer changes between decodes.

Suggestion:

Suggested change

static void d2i_free_prev_key_obj(WOLFSSL_EVP_PKEY* pkey)

#ifdef HAVE_CURVE25519

case WC_EVP_PKEY_X25519:

if (pkey->curve25519 != NULL && pkey->ownCurve25519 == 1) {

wc_curve25519_free(pkey->curve25519);

XFREE(pkey->curve25519, pkey->heap, DYNAMIC_TYPE_CURVE25519);

}

pkey->curve25519 = NULL;

pkey->ownCurve25519 = 0;

break;

#endif

#ifdef HAVE_CURVE448

case WC_EVP_PKEY_X448:

if (pkey->curve448 != NULL && pkey->ownCurve448 == 1) {

wc_curve448_free(pkey->curve448);

XFREE(pkey->curve448, pkey->heap, DYNAMIC_TYPE_CURVE448);

}

pkey->curve448 = NULL;

pkey->ownCurve448 = 0;

break;

#endif

#ifdef HAVE_HKDF

case WC_EVP_PKEY_HKDF:

XFREE(pkey->hkdfSalt, NULL, DYNAMIC_TYPE_SALT);

pkey->hkdfSalt = NULL;

if (pkey->hkdfKey != NULL && pkey->hkdfKeySz > 0) {

ForceZero(pkey->hkdfKey, pkey->hkdfKeySz);

}

XFREE(pkey->hkdfKey, NULL, DYNAMIC_TYPE_KEY);

pkey->hkdfKey = NULL;

XFREE(pkey->hkdfInfo, NULL, DYNAMIC_TYPE_INFO);

pkey->hkdfInfo = NULL;

pkey->hkdfSaltSz = pkey->hkdfKeySz = pkey->hkdfInfoSz = 0;

break;

#endif

Frauschi · 2026-08-03T09:45:13Z

+     * pre-filled the EVP PKEY with the input bytes (d2i_evp_pkey()).
+     * A reused key must be re-populated here. */
+    ret = 1;
+    if (!prePopulated) {


🔵 [Low] Ed25519/Ed448 d2i reuse leaves the stale type and DER in place - pre-existing, but the prePopulated pattern this PR adds is the fix
🔧 NIT bug

This PR fixes reuse for ML-DSA with an explicit prePopulated flag on d2iTryMlDsaKey(), so d2i_evp_pkey() can state "the bytes are already in place" instead of the callee guessing. d2iTryEd25519Key()/d2iTryEd448Key() still use the old heuristic if (*out == NULL || (*out)->pkey.ptr == NULL). On the auto-detect reuse path both conditions are false, so d2i_make_pkey() is skipped entirely and neither pkey->type nor pkey->pkey.ptr is updated.

Confirmed with a PoC - decode an RSA SPKI into a fresh EVP_PKEY, then reuse it for an Ed25519 PKCS#8 via wolfSSL_d2i_PrivateKey_EVP():

after RSA decode : type=16 (RSA=16) pkey_sz=294 after Ed25519 reuse: type=16 (ED25519=2140104148) pkey_sz=294 ed25519=0x101c917d0 rsa=0x101c909b0

The decode reports success, but the object still claims RSA, still serializes the previous RSA DER, and now holds an ed25519 object that wolfSSL_EVP_PKEY_free() will never reach because it switches on type.

This is pre-existing, not a defect of this PR - the PR does not touch either helper, and master (7a8aae3) reproduces the output byte for byte. Raising it here only because this PR establishes the pattern that fixes it.

Suggestion:

Suggested change

if (!prePopulated) {

static int d2iTryEd25519Key(WOLFSSL_EVP_PKEY** out, const unsigned char* mem,

long memSz, int priv, int prePopulated)

{

...

ret = 1;

if (!prePopulated) {

ret = d2i_make_pkey(out, mem, keyIdx, priv, WC_EVP_PKEY_ED25519);

}

/* d2i_evp_pkey_try() passes 0; d2i_evp_pkey()'s typed case passes 1. */

Frauschi · 2026-08-03T09:45:13Z

+         * fails and the fallback public decode must work on a reset key. */
+        {
+            WOLFSSL_EVP_PKEY* spki = NULL;
+            unsigned char der[2048];


🟡 [Medium] New tests add several multi-KB stack buffers
💡 SUGGEST test

Sites: tests/api/test_ossl_x509_pk.c:441,470,501,545; tests/api/test_evp_pkey.c:2735-2737,2790-2792
The new test code declares unsigned char der[2048], unsigned char der[4096], byte pubDer[MLDSA_MAX_PUB_KEY_SIZE + 64] (~2656 bytes), byte draftDer[4096], unsigned char der44[2048], der65[2600], mldsaDer[4096], rsaDer[2048] and rawBlob[2560] as automatic arrays. Several sit in nested scopes inside one function, and compilers are not obliged to overlap nested-scope frames, so test_wolfSSL_X509_set_pubkey can reach ~13 KB of frame on its own. wolfSSL is regularly built and CI-tested with constrained stacks (--enable-smallstack, embedded/RTOS targets, and thread-stack-limited harnesses). The rest of tests/api generally XMALLOCs buffers of this size.

Suggestion:

Suggested change

unsigned char der[2048];

unsigned char* der = NULL;

...

ExpectNotNull(der = (unsigned char*)XMALLOC(4096, NULL,

DYNAMIC_TYPE_TMP_BUFFER));

...

XFREE(der, NULL, DYNAMIC_TYPE_TMP_BUFFER);

OpenSSL compat with ML-DSA

c71c2ba

kojo1 force-pushed the oss_mldsa branch from 5e8b1e3 to c71c2ba Compare July 21, 2026 22:44

kojo1 requested a review from wolfSSL-Bot July 22, 2026 22:47

Frauschi reviewed Jul 24, 2026

View reviewed changes

Comment thread src/x509.c

Frauschi assigned kojo1 Jul 24, 2026

Add ML-DSA test coverage for X509_REQ_sign

d9e04a7

kojo1 force-pushed the oss_mldsa branch from 7d637b2 to d9e04a7 Compare July 24, 2026 21:27

PRIVATE_KEY_UNLOCK to X509 sign/set_pubkey

fa7cda1

kojo1 force-pushed the oss_mldsa branch from 680a71c to 68dd962 Compare July 28, 2026 22:38

Dynamic buff size based on expected cert size

365a9a0

kojo1 force-pushed the oss_mldsa branch from 68dd962 to 365a9a0 Compare July 28, 2026 23:50

kojo1 requested a review from Frauschi July 29, 2026 00:28

kojiws assigned wolfSSL-Bot and kojo1 and unassigned kojo1 and wolfSSL-Bot Jul 29, 2026

Frauschi requested changes Jul 29, 2026

View reviewed changes

kojo1 added 10 commits July 30, 2026 09:32

Explicit prePopulated flag for d2iTryMlDsaKey

8629d9d

Size X509 gen buffer from subject public key too

2c8ce3f

Allow NULL md for ML-DSA X509 signing, add test

5130e00

Single gate for ML-DSA X509 signing capability

f6d88bd

Reset MlDsaKey between decode attempts in X509_set_pubkey

a7d30e2

PKCS#8 algId check and DER-only typed d2i for ML-DSA

aea9b8f

Extend ML-DSA compat test coverage to all levels and untested paths

29887a6

Use mldsa_get_oid_sum for draft-aware OID in X509 signing

7040f83

Consistency cleanups in new ML-DSA X509 code

3cf92c8

Derive WC_MAX_X509_GEN_MLDSA from WC_MAX_X509_GEN

9403fa3

Use canonical ML-DSA names in new X509/EVP code

c363238

kojo1 requested a review from Frauschi July 31, 2026 10:23

Frauschi reviewed Jul 31, 2026

View reviewed changes

kojo1 added 8 commits August 1, 2026 05:59

Test X509_set_pubkey with seed and private-only ML-DSA keys

1115125

Guard ML-DSA key/RNG cleanup in tests against Expect short-circuit

f247487

Reset XFILE handle after every close in ML-DSA tests

4df0330

Free previous algorithm object when d2i repurposes an EVP_PKEY

04063d8

Do not store failed sigType lookup in sigOID

10b876c

Zeroize pkey.ptr before freeing EVP_PKEY key data

bae7b82

Note deliberate md tolerance for ML-DSA X509 signing

1c67473

Fix error code check

50e4d96

kojo1 requested a review from Frauschi August 2, 2026 11:47

Frauschi reviewed Aug 3, 2026

View reviewed changes

dgarske removed the request for review from wolfSSL-Bot August 3, 2026 21:46

-static void d2i_free_prev_key_obj(WOLFSSL_EVP_PKEY* pkey)
+#ifdef HAVE_CURVE25519
+        case WC_EVP_PKEY_X25519:
+            if (pkey->curve25519 != NULL && pkey->ownCurve25519 == 1) {
+                wc_curve25519_free(pkey->curve25519);
+                XFREE(pkey->curve25519, pkey->heap, DYNAMIC_TYPE_CURVE25519);
+            }
+            pkey->curve25519 = NULL;
+            pkey->ownCurve25519 = 0;
+            break;
+#endif
+#ifdef HAVE_CURVE448
+        case WC_EVP_PKEY_X448:
+            if (pkey->curve448 != NULL && pkey->ownCurve448 == 1) {
+                wc_curve448_free(pkey->curve448);
+                XFREE(pkey->curve448, pkey->heap, DYNAMIC_TYPE_CURVE448);
+            }
+            pkey->curve448 = NULL;
+            pkey->ownCurve448 = 0;
+            break;
+#endif
+#ifdef HAVE_HKDF
+        case WC_EVP_PKEY_HKDF:
+            XFREE(pkey->hkdfSalt, NULL, DYNAMIC_TYPE_SALT);
+            pkey->hkdfSalt = NULL;
+            if (pkey->hkdfKey != NULL && pkey->hkdfKeySz > 0) {
+                ForceZero(pkey->hkdfKey, pkey->hkdfKeySz);
+            }
+            XFREE(pkey->hkdfKey, NULL, DYNAMIC_TYPE_KEY);
+            pkey->hkdfKey = NULL;
+            XFREE(pkey->hkdfInfo, NULL, DYNAMIC_TYPE_INFO);
+            pkey->hkdfInfo = NULL;
+            pkey->hkdfSaltSz = pkey->hkdfKeySz = pkey->hkdfInfoSz = 0;
+            break;
+#endif

-    if (!prePopulated) {
+static int d2iTryEd25519Key(WOLFSSL_EVP_PKEY** out, const unsigned char* mem,
+    long memSz, int priv, int prePopulated)
+{
+...
+    ret = 1;
+    if (!prePopulated) {
+        ret = d2i_make_pkey(out, mem, keyIdx, priv, WC_EVP_PKEY_ED25519);
+    }
+/* d2i_evp_pkey_try() passes 0; d2i_evp_pkey()'s typed case passes 1. */

-            unsigned char der[2048];
+unsigned char* der = NULL;
+...
+ExpectNotNull(der = (unsigned char*)XMALLOC(4096, NULL,
+    DYNAMIC_TYPE_TMP_BUFFER));
+...
+XFREE(der, NULL, DYNAMIC_TYPE_TMP_BUFFER);

Uh oh!

Conversation

kojo1 commented Jul 21, 2026

Description

Testing

Checklist

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MemBrowse Memory Report

Uh oh!

Frauschi left a comment

Choose a reason for hiding this comment

🐺 Skoll Code Review

Posted findings

Uh oh!

Uh oh!

kojo1 commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Frauschi commented Jul 27, 2026

Uh oh!

Frauschi left a comment

Choose a reason for hiding this comment

🐺 Skoll Code Review

Posted findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Frauschi left a comment

Choose a reason for hiding this comment

🐺 Skoll Code Review

Posted findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Frauschi left a comment

Choose a reason for hiding this comment

🐺 Skoll Code Review

Posted findings

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jul 21, 2026 •

edited

Loading

kojo1 commented Jul 24, 2026 •

edited

Loading