[Quality-547] Update bert_tiny model to the retrained classifier by evelyn-with-warp · Pull Request #10529 · warpdotdev/warp

evelyn-with-warp · 2026-05-08T23:16:46Z

Description

Simply use retrained classifier to replace the previous one
The new classifier has the same same size as previous one, just further finetuned on prod data with user quickback as misfire groundtruth label
It could resolve the misfires reported in quality-547; For quality 543 on file paths misfires, some are still pending to resolve by improving heuristics.
Technical 1-pager: https://docs.google.com/document/d/192OtIJA_hAQEnLoasgmEXIg0Ge12VbdBEQ4HSiWjpMw/edit?usp=sharing

Linked Issue

The linked issue is labeled ready-to-spec or ready-to-implement.
Where appropriate, screenshots or a short video of the implementation are included below (especially for user-visible or UI changes).

Testing

./script/presubmit

I have manually tested my changes locally with ./script/run

Screenshots / Videos

https://www.loom.com/share/94f99078f8614b2bbc05eab7f8d1a7d1
in this video we tested on the misfires listed in quality-547

left: warp dev
right: warp local build on this change

Video for dogfood feature
https://www.loom.com/share/196ffbc863aa4b39894f4e821ab54d85

Agent Mode

Warp Agent Mode - This PR was created via Warp's AI Agent Mode

Reviewers:

@szgupta @vorporeal

oz-for-oss · 2026-05-08T23:17:00Z

@evelyn-with-warp

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR swaps the bert_tiny.onnx model with a retrained version (bert_tiny_2026May.onnx) to reduce input-classifier misfires reported in Quality-547. The change renames the binary model file and updates the single path reference in Model::model_path(). The model is embedded at compile time via RustEmbed, and no other code references the old filename.

Concerns

The model file rename bakes the training date into the filename (bert_tiny_2026May.onnx). This is fine as a one-off, but if models are swapped regularly a versioning scheme (e.g. a monotonic version number or hash suffix) would be easier to maintain. This is non-blocking.
The PR description sentence "Simply use retrained classifier to replace the previously" appears truncated. Minor, but worth fixing for posterity.

Security

No security concerns. The change swaps a binary model asset and updates a static path literal. No new inputs, dependencies, secrets, or network surfaces are introduced.

Verdict

Found: 0 critical, 0 important, 1 suggestions

Approve

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss · 2026-05-08T23:20:53Z

    fn model_path(&self) -> &'static str {
        match self {
-            Model::BertTiny => "bert_tiny.onnx",
+            Model::BertTiny => "bert_tiny_2026May.onnx",


💡 [SUGGESTION] Consider a versioning scheme that doesn't encode calendar dates (e.g. bert_tiny_v2.onnx). Date-suffixed names become ambiguous when multiple retrains happen in the same month and make it harder to correlate with training metadata.

@szgupta @vorporeal is this a real concern (as we won't refresh this often)? if so, could you suggest proper model versioning strategy for client app?

Let's just go with v2

szgupta

@evelyn-with-warp we should gate this to dogfood. In this case, that means not including the new model in preview / stable release binaries so that we don't bloat the binary size unnecessarily.

evelyn-with-warp · 2026-05-11T15:54:15Z

@evelyn-with-warp we should gate this to dogfood. In this case, that means not including the new model in preview / stable release binaries so that we don't bloat the binary size unnecessarily.

sg; the following change on heuristics will be also bundled in the same feature flag in dogfooding

Address review feedback to use a version suffix instead of a date-encoded name, avoiding ambiguity when multiple retrains land in the same month. Co-Authored-By: Oz <oz-agent@warp.dev>

The previous pattern `input_classifier/models/*` did not match the actual path `crates/input_classifier/models/*`, so the model binaries were stored as raw blobs in git history. Update to `crates/input_classifier/models/**` and explicitly opt the tokenizer JSON back out of LFS so it stays human-diffable. Co-Authored-By: Oz <oz-agent@warp.dev>

- Add bert_tiny.onnx model under crates/input_classifier/models/onnx - Wire up nld_onnx_model_v2 feature flag in warp_features - Update input_classifier to load the v2 model when the flag is enabled - Bootstrap script: ensure git-lfs pulls the model binaries Co-Authored-By: Oz <oz-agent@warp.dev>

Co-Authored-By: Oz <oz-agent@warp.dev>

evelyn-with-warp · 2026-05-11T18:00:56Z

  fi
 }

+# Ensure Git LFS is installed and initialized for the repo. The


might or might not need this as our model is not large (17 MB)
no harm to add to be defensive;
pls lmk if it's better not to do this @szgupta

evelyn-with-warp · 2026-05-11T18:01:45Z

 *.pdb filter=lfs diff=lfs merge=lfs -text
-input_classifier/models/* filter=lfs diff=lfs merge=lfs -text
-input_classifier/models/**/*tokenizer.json linguist-generated=true
+crates/input_classifier/models/** filter=lfs diff=lfs merge=lfs -text


the prev path did not work (so git cannot detect binary file change and just report it as rename, which was confusing)

evelyn-with-warp · 2026-05-11T18:05:55Z

note that the change reported on binary model files are due to we changed it from raw blob to Git LFS pointer

evelyn-with-warp · 2026-05-12T20:29:15Z

here's new PR #10756 for the same purpose

oz-for-oss · 2026-05-12T20:29:28Z

@evelyn-with-warp

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR adds a second BERT-tiny ONNX classifier, feature-flagged selection logic, Git LFS setup for classifier model binaries, and related bootstrap/documentation updates.

Concerns

The rollout is not isolated to the new v2 path: crates/input_classifier/models/onnx/bert_tiny.onnx is also modified, so users with v2 disabled may still receive a changed baseline model instead of a stable fallback.
crates/input_classifier/models/fasttext/cmd_lang_classifier_v4.bin changes in this PR even though the description and rollout logic focus on the ONNX BERT-tiny classifier; please either justify and test the fasttext change or revert it.
The inline comment calls out that adding nld_onnx_model_v2 to nld_improvements can enable v2 outside the intended dogfood rollout.

Verdict

Found: 0 critical, 3 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss · 2026-05-12T20:37:50Z

 vim_code_editor = []
 allow_opening_file_links_using_editor_env = []
-nld_improvements = ["nld_onnx_model"]
+nld_improvements = ["nld_onnx_model", "nld_onnx_model_v2"]


⚠️ [IMPORTANT] Adding nld_onnx_model_v2 to nld_improvements means any build that already enables this meta-feature will auto-enable NLDOnnxModelV2Enabled through enabled_features, so v2 is no longer dogfood-only; keep this meta-feature on the baseline model until the wider rollout is intentional.

Update bert_tiny model to the retrained classifier

fea167f

cla-bot Bot added the cla-signed label May 8, 2026

oz-for-oss Bot reviewed May 8, 2026

View reviewed changes

szgupta reviewed May 11, 2026

View reviewed changes

evelyn-with-warp and others added 5 commits May 11, 2026 08:55

Rename bert_tiny_2026May.onnx to bert_tiny_v2.onnx

a147fe6

Address review feedback to use a version suffix instead of a date-encoded name, avoiding ambiguity when multiple retrains land in the same month. Co-Authored-By: Oz <oz-agent@warp.dev>

Merge branch 'master' into evelyn/quality-543

dd73997

Merge branch 'evelyn/dogfood-nld-classifier-v2' into evelyn/quality-543

d07877c

Co-Authored-By: Oz <oz-agent@warp.dev>

evelyn-with-warp commented May 11, 2026

View reviewed changes

evelyn-with-warp requested a review from szgupta May 11, 2026 19:07

evelyn-with-warp closed this May 12, 2026

evelyn-with-warp reopened this May 12, 2026

evelyn-with-warp closed this May 12, 2026

oz-for-oss Bot reviewed May 12, 2026

View reviewed changes

Conversation

evelyn-with-warp commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Linked Issue

Testing

Screenshots / Videos

Agent Mode

Reviewers:

Uh oh!

oz-for-oss Bot commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Security

Verdict

Uh oh!

oz-for-oss Bot May 8, 2026

Choose a reason for hiding this comment

Uh oh!

evelyn-with-warp May 9, 2026

Choose a reason for hiding this comment

Uh oh!

szgupta May 11, 2026

Choose a reason for hiding this comment

Uh oh!

szgupta left a comment

Choose a reason for hiding this comment

Uh oh!

evelyn-with-warp commented May 11, 2026

Uh oh!

evelyn-with-warp May 11, 2026

Choose a reason for hiding this comment

Uh oh!

evelyn-with-warp May 11, 2026

Choose a reason for hiding this comment

Uh oh!

evelyn-with-warp commented May 11, 2026

Uh oh!

evelyn-with-warp commented May 12, 2026

Uh oh!

oz-for-oss Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

oz-for-oss Bot May 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

evelyn-with-warp commented May 8, 2026 •

edited

Loading

oz-for-oss Bot commented May 8, 2026 •

edited

Loading

oz-for-oss Bot commented May 12, 2026 •

edited

Loading