Skip to content

feat(agent): classify model families#2525

Closed
cyq1017 wants to merge 1 commit into
Hmbown:mainfrom
cyq1017:codex/2081-model-family
Closed

feat(agent): classify model families#2525
cyq1017 wants to merge 1 commit into
Hmbown:mainfrom
cyq1017:codex/2081-model-family

Conversation

@cyq1017
Copy link
Copy Markdown
Contributor

@cyq1017 cyq1017 commented Jun 1, 2026

Problem
Model-family identity needs a shared agent-crate primitive before TUI, desktop, and runtime API surfaces can render consistent model affordances.

Change

  • Add ModelFamily to codewhale-agent.
  • Add model_family(model_id) classification for common first-party, routed, and self-hosted model ids.
  • Keep unknown/custom gateway ids in the inferencer bucket.

Verification

  • cargo test -p codewhale-agent model_family --locked
  • rustfmt crates/agent/src/lib.rs --edition 2024 --check
  • cargo test -p codewhale-agent --locked
  • git diff --check

Partially addresses #2081.

Greptile Summary

This PR introduces a ModelFamily enum and a model_family() classification function to codewhale-agent, providing a shared primitive for identifying model families across TUI, desktop, and runtime API surfaces. Classification is done via substring/prefix matching on a normalized (trimmed, lowercased) model ID string.

  • Adds 11 ModelFamily variants covering major first-party and self-hosted model providers, with an Inferencer fallback for unknown or custom gateway IDs.
  • model_family() applies ordered substring checks, correctly prioritising GptOss before the broader gpt-/openai/ OpenAI check.
  • Tests cover known model IDs, routed gateway forms, and the Inferencer fallback, but bare OpenAI o-series IDs (o1-mini, o3, o4-mini) are not handled and silently land in Inferencer.

Confidence Score: 3/5

Mostly safe to merge, but bare OpenAI o-series model IDs are silently misclassified as Inferencer rather than OpenAI, which will produce wrong affordances in any client that consumes this function.

The o1/o3/o4 classification gap is a real behavioral defect: any caller passing a bare o1-mini, o3, or o4-mini ID will receive Inferencer instead of OpenAI. Since this function is being introduced precisely to drive UI affordances across multiple surfaces, misclassifying a prominent and widely-used model family will have visible downstream impact the moment a client consumes it.

crates/agent/src/lib.rs — specifically the OpenAI matching block in model_family() and the missing o-series test coverage.

Important Files Changed

Filename Overview
crates/agent/src/lib.rs Adds ModelFamily enum and model_family() classifier with substring/prefix matching; OpenAI o-series bare IDs (o1, o3, o4) fall through to Inferencer, and #[must_use] is placed before the doc comment rather than after it.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A([model_id: &str]) --> B[normalize: trim + lowercase]
    B --> C{is_empty?}
    C -- yes --> Z([Inferencer])
    C -- no --> D{contains 'deepseek'?}
    D -- yes --> E([DeepSeek])
    D -- no --> F{contains 'claude' or 'anthropic'?}
    F -- yes --> G([Anthropic])
    F -- no --> H{contains 'gpt-oss' or 'gpt_oss'?}
    H -- yes --> I([GptOss])
    H -- no --> J{starts_with 'gpt-' OR contains '/gpt-' OR 'openai/'?}
    J -- yes --> K([OpenAI])
    J -- no --> L{contains 'gemini', 'gemma', or 'google/'?}
    L -- yes --> M([Google])
    L -- no --> N{contains 'llama', 'meta-', or 'meta/'?}
    N -- yes --> O([Meta])
    N -- no --> P{contains 'mistral', 'mixtral', or 'codestral'?}
    P -- yes --> Q([Mistral])
    P -- no --> R{contains 'qwen'?}
    R -- yes --> S([Qwen])
    R -- no --> T{contains 'grok'?}
    T -- yes --> U([Grok])
    T -- no --> V{contains 'cohere' or 'command-r'?}
    V -- yes --> W([Cohere])
    V -- no --> Z
Loading

Fix All in Codex Fix All in Claude Code Fix All in Cursor

Reviews (1): Last reviewed commit: "feat(agent): classify model families" | Re-trigger Greptile

Greptile also left 2 inline comments on this PR.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a ModelFamily enum and a model_family helper function to classify model IDs into their respective families (such as OpenAI, Google, Meta, and Mistral), along with comprehensive unit tests. The review feedback suggests several improvements to the classification logic: removing a redundant check for "/openai/" since "openai/" already covers it, adding a check for "gemma" to correctly classify Google's open-weights models, and adding checks for "mixtral" and "codestral" to ensure Mistral AI's other prominent models are not incorrectly classified as Inferencer.

Comment thread crates/agent/src/lib.rs
Comment on lines +350 to +354
if normalized.starts_with("gpt-")
|| normalized.contains("/gpt-")
|| normalized.contains("openai/")
|| normalized.contains("/openai/")
{
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The check normalized.contains("/openai/") is redundant because any string containing "/openai/" will also match normalized.contains("openai/"). Removing it simplifies the condition.

    if normalized.starts_with("gpt-")
        || normalized.contains("/gpt-")
        || normalized.contains("openai/")
    {

Comment thread crates/agent/src/lib.rs Outdated
Comment on lines +357 to +359
if normalized.contains("gemini") || normalized.contains("google/") {
return ModelFamily::Google;
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Google's open-weights model family Gemma (e.g., gemma-2-9b) is widely used, especially in local setups (like Ollama) where the google/ prefix might be omitted. Adding a check for "gemma" ensures these models are correctly classified under the Google family.

    if normalized.contains("gemini")
        || normalized.contains("gemma")
        || normalized.contains("google/")
    {
        return ModelFamily::Google;
    }

Comment thread crates/agent/src/lib.rs Outdated
Comment on lines +364 to +366
if normalized.contains("mistral") {
return ModelFamily::Mistral;
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Mistral AI's model family includes prominent models like Mixtral (e.g., mixtral-8x7b) and Codestral (e.g., codestral-22b). Since these names do not contain the substring "mistral", they will currently fall back to Inferencer. Adding checks for "mixtral" and "codestral" ensures they are correctly classified.

Suggested change
if normalized.contains("mistral") {
return ModelFamily::Mistral;
}
if normalized.contains("mistral")
|| normalized.contains("mixtral")
|| normalized.contains("codestral")
{
return ModelFamily::Mistral;
}

@cyq1017 cyq1017 force-pushed the codex/2081-model-family branch from 874f22a to fff82bc Compare June 1, 2026 16:57
@cyq1017 cyq1017 force-pushed the codex/2081-model-family branch from fff82bc to bfcbe9d Compare June 1, 2026 16:58
Comment thread crates/agent/src/lib.rs
Comment on lines +564 to +566
#[must_use]
/// Classify a model identifier by its underlying model family.
pub fn model_family(model_id: &str) -> ModelFamily {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 #[must_use] placed before the doc comment

In Rust, the conventional (and rustfmt-enforced) order is doc comment first, then outer attributes. Placing #[must_use] before /// can cause rustfmt --edition 2024 --check to report a diff on some toolchain versions and diverges from the pattern used by every other documented item in this file.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Codex Fix in Claude Code Fix in Cursor

Comment thread crates/agent/src/lib.rs
Comment on lines +581 to +586
if normalized.starts_with("gpt-")
|| normalized.contains("/gpt-")
|| normalized.contains("openai/")
{
return ModelFamily::OpenAI;
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 OpenAI o-series models not matched with bare IDs

o1-mini, o1-preview, o3, o3-mini, o4-mini and related IDs don't start with gpt-, don't contain /gpt-, and don't contain openai/, so model_family("o1-mini") currently returns Inferencer. They're reachable via the openai/o1-mini routed form, but direct/bare invocations silently fall through. Adding normalized.starts_with("o1") || normalized.starts_with("o3") || normalized.starts_with("o4") (or similar) alongside the existing gpt- prefix check would close this gap.

Fix in Codex Fix in Claude Code Fix in Cursor

@Hmbown
Copy link
Copy Markdown
Owner

Hmbown commented Jun 3, 2026

Cherry-picked to codex/v0.8.53 in 4556282.

@Hmbown Hmbown closed this Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants