feat!: Add AI online evaluations (Judge, Evaluator, IRunner)

[mattrmc1]

BEGIN_COMMIT_OVERRIDE
feat: Add AI online evaluations (Judge, Evaluator, IRunner)
feat!: JudgeResult defaults for sampled and success changed from true to false per AIEVALS spec
END_COMMIT_OVERRIDE

Summary

Adds AI online evaluations to LaunchDarkly.ServerSdk.Ai. A caller that supplies a runnerFactory to LdAiClient gets automatic judge evaluation wired into every CompletionConfig and AgentConfig — each returned config carries an Evaluator that can score model output against the judges declared in the flag's judgeConfiguration. When no runnerFactory is provided (or no judges are configured), configs receive a noop Evaluator so callers never need null checks.

Implements the AIEVALS and AIRUNNER specs (sections 1.1–1.4). createJudge (AIEVALS 1.2) is intentionally omitted per .NET/Java convention — judges are created internally by the SDK, not by user code.

New public types

// Provider-facing runner interface (AIRUNNER 1.2)
public interface IRunner
{
    Task<RunnerResult> RunAsync(string input,
        IReadOnlyDictionary<string, object> outputType = null);
}

// Runner return type (AIRUNNER 1.3)
public sealed record RunnerResult(
    string Content,
    AiMetrics Metrics,
    object Raw = null,
    IReadOnlyDictionary<string, object> Parsed = null);

// Evaluation orchestrator (AIEVALS 1.4)
public sealed class Evaluator
{
    public static Evaluator Noop();
    public Task<IReadOnlyList<JudgeResult>> EvaluateAsync(string input, string output);
}

// Single-judge executor (AIEVALS 1.1)
public sealed class Judge
{
    public LdAiJudgeConfig Config { get; }
    public IRunner Runner { get; }
    public Task<JudgeResult> EvaluateAsync(string input, string output, double? samplingRate = null);
    public Task<JudgeResult> EvaluateMessagesAsync(
        IReadOnlyList<LdAiConfigTypes.Message> messages,
        RunnerResult runnerResult, double? samplingRate = null);
}

LdAiClient changes

// New optional parameter
public LdAiClient(ILaunchDarklyClient client,
    Func<LdAiJudgeConfig, IRunner> runnerFactory = null);

When runnerFactory is non-null, ConfigFactory.BuildEvaluator iterates the flag's judgeConfiguration, evaluates each judge key as a flag variation, creates a Judge + IRunner pair per enabled judge, and attaches the resulting Evaluator to the config. Disabled judges, null runners, and initialization exceptions are logged and skipped — no single judge failure prevents the others from being built.

LdAiConfig base class

All config types (LdAiCompletionConfig, LdAiAgentConfig, LdAiJudgeConfig) now carry an Evaluator property via the base class. LdAiJudgeConfig always receives Evaluator.Noop() (judges don't evaluate themselves).

JudgeResult changes

Updated to match AIEVALS 1.3.1 defaults:

Field	Before	After
MetricKey	required	optional (default null)
Score	required	optional (default 0.0)
Sampled	default true	default false
Success	default true	default false
ErrorMessage	—	new field
Reasoning	—	new field

Null safety (ref: java-core#175 discussion)

BuildEvaluator handles every failure mode without throwing:

• runnerFactory == null or empty judgeConfiguration → Evaluator.Noop()
• Runner factory returns null → warn + skip judge
• Judge config disabled → warn + skip judge
• Any exception during judge init → warn + skip judge
• Missing evaluationMetricKey → JudgeResult(success: false, errorMessage: ...)
• Score out of [0, 1] range → JudgeResult(success: false, errorMessage: ...)
• Sampling rate NaN/Infinity/negative/> 1.0 → normalized to safe bounds

Judge constructor still uses ArgumentNullException guards, but these are never hit in practice because BuildEvaluator validates inputs before construction.

Migration

None required. The LdAiClient constructor gains an optional runnerFactory parameter (default null) — existing callers are unaffected. JudgeResult default changes are source-compatible (all parameters are now optional with safe defaults). No members removed or renamed.

Test plan

• dotnet test pkgs/sdk/server-ai/test/LaunchDarkly.ServerSdk.Ai.Tests.csproj --framework net8.0 passes
• JudgeTest (435 lines) covers: successful evaluation with score/reasoning extraction, sampling skip path, samplingRate edge cases (NaN, negative, > 1.0), runner exception handling, missing evaluationMetricKey validation, out-of-range score rejection with errorMessage, evaluateMessages formatting, null/empty message handling
• EvaluatorTest (210 lines) covers: noop returns empty list, multi-judge execution with per-judge sampling, missing judge key logs warning and skips, noop does not log warnings
• LdAiCompletionConfigTest (189 lines added) covers: Evaluator attached when runnerFactory provided, noop Evaluator when no runner factory, noop when judgeConfiguration is empty, disabled judge skipped, null runner skipped
• LdAiJudgeConfigTest covers: JudgeResult ErrorMessage and Reasoning fields, judge config always receives noop Evaluator
• LdAiConfigTrackerTest covers: TrackJudgeResult with new optional fields

---

Note

Medium Risk
New public API and config-build path that performs extra flag evaluations and optional model calls via IRunner; failures are mostly logged/skipped but behavior and JudgeResult default changes affect metrics emission semantics.

Overview
Adds AI online evaluations to the server AI SDK: completion and agent configs can carry a non-null Evaluator that runs configured judges against model input/output when callers supply an optional runnerFactory on LdAiClient.

Wiring: ConfigFactory.BuildEvaluator resolves each judge key via flag variation, builds Judge + IRunner pairs (skipping disabled judges, null runners, and init failures), and attaches the result to normal and default fallback paths for completion/agent configs. Agent graph builds pass graphKey into judge trackers. Without a factory or judges, configs get Evaluator.Noop().

New surface: IRunner, RunnerResult, Judge (structured score/reasoning via runner, sampling, metrics tracking), and Evaluator.EvaluateAsync (does not call TrackJudgeResult—callers emit events).

Spec-aligned tweaks: JudgeResult defaults Sampled/Success to false; optional ErrorMessage/Reasoning. Judge samplingRate is nullable (omit = 100%); parsing and default ToLdValue only serialize rate when set.

^{Reviewed by Cursor Bugbot for commit 636858c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Judge sampling ignores interpolate flag

Medium Severity

BuildEvaluator always calls BuildJudgeConfig with the default interpolate: true, even when the parent config was built with interpolate: false (template APIs). Nested judge configs then run Mustache interpolation (including ldctx) while completion/agent instructions and messages stay as raw templates, so template configs and runnerFactory output can disagree with the documented template behavior.

Additional Locations (2)

• pkgs/sdk/server-ai/src/Config/ConfigFactory.cs#L267-L308
• pkgs/sdk/server-ai/src/LdAiClient.cs#L156-L165

 

^{Reviewed by Cursor Bugbot for commit 6f7d88d. Configure here.}

[mattrmc1]

Not a bug. Attached judges are operational -- they execute against a model provider via IRunner.RunAsync and require interpolated prompts to function. A judge receiving raw {{variable}} tokens would produce nonsense. When fetching a judge directly with interpolation: false, the raw {{variable}} tokens will return correctly

[mattrmc1]

Messages when fetching toxicity judge as template:

[
    {
        "content": "You are a business information accuracy and safety expert. Evaluate....",
        "role": "system"
    },
    {
        "content": "MESSAGE HISTORY:\n{{message_history}}\n",
        "role": "assistant"
    },
    {
        "content": "RESPONSE TO EVALUATE:\n{{response_to_evaluate}}\n",
        "role": "user"
    }
]

[cursor]

Duplicate judge keys run twice

Low Severity

If judgeConfiguration.judges lists the same key more than once, BuildEvaluator stores one Judge in a dictionary but builds filteredConfig from the full list without deduplicating. EvaluateAsync then invokes the same judge once per duplicate entry and returns multiple results for one logical judge.

Additional Locations (1)

• pkgs/sdk/server-ai/src/Config/ConfigFactory.cs#L305-L307

 

^{Reviewed by Cursor Bugbot for commit 6f7d88d. Configure here.}

[mattrmc1]

Is this actually a bug? Should the SDK be defensive about this or faithfully evaluate the list it receives (even if duplicates are present)?

cc @jsonbailey

[cursor]

Shared Random not thread-safe

Medium Severity

Evaluator stores one Random instance and passes it into every concurrent Judge.EvaluateAsync call for sampling. System.Random is not thread-safe, so parallel EvaluateAsync on the same config can corrupt sampling state and yield incorrect skip/run decisions or unstable behavior under load.

 

^{Reviewed by Cursor Bugbot for commit 2562911. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 4 total unresolved issues (including 3 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 636858c. Configure here.}

[cursor]

Template fetch builds live evaluators

Medium Severity

CompletionConfigTemplate and AgentConfigTemplate call the factory with interpolate: false, but BuildCompletionConfig and BuildAgentConfig always invoke BuildEvaluator regardless of that flag. With a non-null runnerFactory, template retrieval still evaluates judge flags, interpolates judge prompts, and attaches a runnable Evaluator while completion/agent content stays un-interpolated.

Additional Locations (2)

• pkgs/sdk/server-ai/src/Config/ConfigFactory.cs#L152-L153
• pkgs/sdk/server-ai/src/Config/ConfigFactory.cs#L100-L101

 

^{Reviewed by Cursor Bugbot for commit 636858c. Configure here.}