Ci interop fixes

[Frauschi]

Summary

The nightly Interop workflow had two jobs (cisco/libest and smallstep/step-ca) that were green but testing nothing: their third-party dependency failed to build under continue-on-error, so the interop script found no binary in PATH and exited 77 (a neutral skip). This PR fixes both dependency builds, makes the two scripts exercise their peers for real, and cleans up the SCEP recipient handling that the step-ca work exposed.

Net effect: 2 of 5 interop targets go from decorative to actually validating wolfCert against a real implementation.

What changed

CI dependency builds (.github/workflows/interop.yml)

• cisco/libest is archived and written for OpenSSL 1.x. On the OpenSSL 3.0 runner its build aborted twice: configure now demands an explicit safec choice, and linking fails because OpenSSL 3.0 removed FIPS_mode() / FIPS_mode_set(), which libest still calls. Added --disable-safec and a force-included compat header that expands both symbols to no-ops (defining the function-like macros in a header rather than via -D keeps their parens off gcc's command line).
• smallstep/step-ca cgo-links libpcsclite (PC/SC, for its PIV/YubiKey provisioners); install libpcsclite-dev before go install, or the build aborts at pkg-config.

libest D2 now enrolls for real (tests/interop/est_libest.sh)

The libest estclient -> wolfcert-server direction was soft-skipping: it guessed estclient's trust-anchor flag (--pem-file / -c) and treated -o as a cert file. Use the real flags - --trustanchor for the server anchor, -o as the output directory - let estclient auto-generate its RSA key, then base64-decode the emitted cert-*.pkcs7 and assert wolfcert-server issued the requested CN.

step-ca gets an RSA SCEP chain (tests/interop/est_stepca.sh)

step-ca's default ECDSA CA cannot be a SCEP decrypter (RFC 8894 is RSA-only). After step ca init, swap in an RSA root + intermediate (per Smallstep's own guidance) so the intermediate step-ca uses as the SCEP decrypter - and every cert GetCACert returns - is RSA, and set the provisioner to AES-128-CBC for the CertRep. [2] is now a strict assertion.

wolfCert: clearer error for a non-RSA RA cert (src/scep/scep_msg.c, tests/unit/test_scep_msg.c)

wolfcert_scep_envelop encrypts the pkcsPKIEnvelope to the RA/CA public key with CMS key transport, which wolfSSL only supports for an RSA recipient. An ECC RA cert (like step-ca's default chain) previously fell through to wolfSSL's key-agreement path and failed deep in the encoder with BAD_KEYWRAP_ALG_E (-239). Check p7->publicKeyOID right after InitWithCert and return WOLFCERT_ERR_UNSUPPORTED with an actionable message. Added a test_scep_msg negative test: a self-signed ECC RA cert is rejected, an RSA one still envelops.

docs (docs/INTEROP.md)

New document the scripts already referenced: the interop targets, the best-effort 77-skip convention (and the caveat that a broken dependency build masquerades as a skip), the libest/step-ca dependency-build notes, and notes 1-6.

Testing

Verified via workflow_dispatch runs of interop.yml on the fork:

• cisco/libest - builds clean; both directions PASS ([1] wolfcert-client -> estserver, [2] estclient -> wolfcert-server).
• smallstep/step-ca - builds clean; the RSA chain swap works end-to-end (the failure moved from -239 BAD_KEYWRAP_ALG_E to -229, i.e. the pkcsPKIEnvelope now encrypts to step-ca's RSA key and step-ca issues the cert).
• test_scep_msg (local ctest) - new ECC-rejection test passes; full suite 24/24 green.

Known-red jobs (self-heal on a pending wolfSSL fix)

Two SCEP jobs stay red until wolfSSL PR #10928 (pkcs7_fix) reaches master, then self-heal to PASS:

• micromdm/scep D2 - unchanged; already tracked.
• smallstep/step-ca [2] - after the RSA swap, its only remaining failure is verifying the CertRep signature (VerifySignedData / ASN_SIG_CONFIRM_E, -229). step-ca's SCEP is built on github.com/smallstep/scep (the micromdm library), so it hits the exact same PKCS#7 signature gap as micromdm. Kept strict, consistent with the micromdm interop.

Both are documented in docs/INTEROP.md notes 3 and 6.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11

Scan targets checked: wolfcert-bugs, wolfcert-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11

Scan targets checked: wolfcert-bugs, wolfcert-src

No new issues found in the changed files. ✅