API Design: Factory Pattern Friction With Rust

[officialfrancismendoza]

Problem

Rust is a multi-paradigm systems programming language that is not purely object-oriented. In comparison to other Bouncy Castle cryptography libraries that are expressed in native object-oriented languages (C# and Java, for example), Rust is unique given its strict nature.

As part of API design, factories are a contentious issue. In C# and Java counterparts, they are a utility class that creates cryptographic objects from encoded data without the caller needing to know the exact implementation type, which follows a factory pattern. They both use runtime data-driven dispatch (read and parse metadata, find discriminator field, choose constructor based on that value) to determine what specific cryptographic constructor to use.

Rust does not naturally model this pattern in the same way as Java or C#. As part of API design, we need to determine how factory-style decoding should be represented:

• Enum-based return types
• Trait objects
• Algorithm-specific constructors only
• Some hybrid approach

The goal is to provide a Rust-idiomatic API while preserving support for decoding algorithm-agnostic encoded data.

[mcr]

My experience is with Ruby (duck typing), C, and Rust. I do tolerate Java, and have never done C#.
I have seen the Factory pattern all over Java, and I've seldom understood what it was really doing for me.
In Rust, we have a pattern of having "Builders", and I don't find that pattern particularly exciting.
What I see in many places is that either the author of the application knows EXACTLY what they want, and they spend 50 lines of code convincing the Factory or Builder about this. Or, they either just don't care (give me the best default).
Added to this is various levels of nonsense related to how some operator/user of the application wants to configure things.
(The OpenSSL provider interface is very half-baked, has surprising limits/complexities)

The place where I think this is MOST visible is around private keys/signing. Because the source of the private key is increasingly not a file in a magic directory, but some HSM-like thing (TEE, TPM, USB/Yubico, TCP-connection-to-HSM-appliance).
The pattern that I think is best is dependency injection via "dyn trait".
This pattern is pretty much mandatory in Rust to enable sensible testing, and one sees this in the way that Rng are passed in.
OTH, I really hate the limitations that dyn trait creates, causing one to Box<> things one might not have, or resulting in untypable (like, with a keyboard...) type names.
I hate that I wind up with a dyn trait JUST so that I can write test cases, but any real application is going to pick one thing. Maybe LTO can make me feel less wasteful, but ...

An ENUM is not easily extendable outside of a package. So, an ENUM does not solve some new signer/private-key-holder that comes later. And one gets into feature-hell.

What is "algorithm-agnostic encoded data"? By algorithm, does this refer to "MLDSA" as an example?

[officialfrancismendoza]

@mcr good question and thanks for the input. By "algorithm-agnostic encoded data", I mean encoded data where the caller doesn't know the exact concrete algorithm-specific type at the call site.

Example: the input may contain metadata (an algorithm identifier, OID, parameter set, other discriminator field). Decoding layer would then read the metadata, parse it, and determine what algorithm-specific implementation to select.

MLDSA would be a great example of an algorithm. In the library, every algorithm is its own crate in bc-rust. The caller may have “some encoded public key/private key/signature-related object,” but not know up front whether it is ML-DSA, RSA, ECDSA, Ed25519, etc.

In Java/C#, this is usually modeled as a factory: parse the metadata, choose the concrete implementation, and return a common supertype/interface. In Rust, the design question is what the equivalent return shape should be: an enum, a dyn Trait, algorithm-specific constructors only, or some hybrid.

I agree that enums are not ideal for externally extensible cases, because downstream crates cannot add new variants, which is bad API design. They are good when the supported algorithm set is intentionally closed, but less good for provider/HSM/plugin-style use cases.

For private-key/signing specifically, I also think dyn Trait is hard to avoid if we want algorithm-agnostic use, testability, and support for keys that may live behind HSMs, TPMs, TEEs, YubiKeys, or remote signing services. The tradeoff is the usual Rust cost: object safety constraints, boxing/lifetimes, and less precise static typing. But for this class of API, will have to think about it. @ounsworth, any thoughts here?