fix(protect): align metric names across Python and TypeScript SDKs (TH-7040)

[KarthikAvinashFI]

Why

The two Protect SDKs exposed incompatible metric names for the same guardrail rules, so a rule config authored in one language was rejected by the other. The wire protocol itself was fine (SDK payloads carry the template's numeric eval_id, not the name), but the SDK-level validation rejected foreign names.

Second issue: TypeScript's metric_map shipped with Tone as a first-class metric. Protect does not support Tone.

What changed

Both SDKs now accept the same superset: 4 canonical snake_case metrics plus every historical alias, each resolving to the correct template.

Metric	Type	Template
toxicity	canonical	Toxicity
bias_detection	canonical	BiasDetection
prompt_injection	canonical	PromptInjection
data_privacy_compliance	canonical	DataPrivacyCompliance
content_moderation	legacy (originally Python-side)	Toxicity (warns)
security	legacy (originally Python-side)	PromptInjection (warns)
Toxicity	legacy (originally TS-side)	Toxicity (warns)
Sexism	legacy (originally TS-side)	BiasDetection (warns)
Prompt Injection	legacy (originally TS-side)	PromptInjection (warns)
Data Privacy	legacy (originally TS-side)	DataPrivacyCompliance (warns)
Tone	removed	not supported

Legacy aliases stay supported and emit a one-shot deprecation warning per process. Removal is deferred to a future release.

Files touched

• python/fi/evals/protect.py
  • metric_map extended with the TS-side legacy names so cross-language configs resolve; _deprecated_metrics extended in step so each new alias emits a FutureWarning pointing at the canonical name.
  • Replace string equality check rule["metric"] == "Data Privacy" with identity check template_class is DataPrivacyCompliance so every alias resolving to that template picks up check_internet=False. See "Behaviour change" below.
  • Remove verified-dead is_tone_metric branch (validation at line 527 rejected the metric name before this branch could fire).
  • Remove now-unused imports (Sexist, Tone) and valid_types set.
  • use_flash fallback default and docstring example use canonical toxicity instead of deprecated content_moderation.
• python/fi/evals/guardrails/backends/turing.py
  • TuringBackend.classify was hard-coding the deprecated content_moderation and security names, which would have emitted a FutureWarning on every classify call after the alias table landed. Switched to canonical toxicity and prompt_injection. The downstream category-lookup dict keeps the legacy keys so any external caller that still passes those names as rules continues to categorize correctly.
• typescript/ai-evaluation/src/protect.ts
  • Rewrite metric_map to include the 4 canonical names plus every legacy alias.
  • Add module-level DEPRECATED_METRICS + one-shot console.warn per alias per process.
  • Replace rule.metric === "Data Privacy" with templateInfo === Templates.DataPrivacyCompliance.
  • Remove the Tone validation branch and the validTypes set.
  • useFlash fallback default uses canonical toxicity.

Behaviour change on Python's data-privacy path

The pre-fix Python code compared rule["metric"] == "Data Privacy" against a value that was never present in Python's metric_map, so check_internet=False was effectively never set on any real Python request. The identity check now correctly fires for every alias resolving to DataPrivacyCompliance, which matches how the TypeScript SDK has behaved all along.

Practical effect: Python callers running a data_privacy_compliance rule will now send check_internet=False in the config payload. This aligns with the intended semantics of Protect (guardrails should evaluate the input in isolation) and with the TypeScript SDK. Anyone who relied on the old bug will need to opt in to internet lookups explicitly.

Backend compatibility

The wire payload carries template.eval_id (a numeric ID), so metric-name divergence between SDK and platform BE is never seen at the API boundary. The BE also has ee/protect/helper.py::_UI_TO_METRIC which normalises many name variants on its own surface (dashboard / config UI), but that surface is decoupled from the SDK's /eval request path. Nothing on the BE needs to change for this PR.

Tests

• python/tests/evals/local/test_protect_metric_map.py (new, 13 tests): pins each canonical mapping, each of the 6 legacy alias mappings (both Python-legacy and TS-legacy), the deprecation warning fires for aliases, no warning for canonicals, and Tone is rejected.
• typescript/ai-evaluation/src/__tests__/protect.test.ts (extended): parametrised coverage for canonical + all 6 legacy aliases, plus a regression asserting Tone is now rejected. Updated one pre-existing test to match the current TS error message (was aspirational; already failing on dev).

Known parity gap (out of scope)

The TS protect() signature accepts string only; Python accepts string | list[str]. Widening the TS surface is a separate change and not addressed here.

Commands

# Python
cd python && python -m pytest tests/evals/local/test_protect_metric_map.py -v

# TypeScript
cd typescript/ai-evaluation && pnpm test src/__tests__/protect.test.ts