Skip to content

feat: add schema-compare command to test harness#2891

Draft
markphelps wants to merge 1 commit intomainfrom
schema-compare-harness
Draft

feat: add schema-compare command to test harness#2891
markphelps wants to merge 1 commit intomainfrom
schema-compare-harness

Conversation

@markphelps
Copy link
Copy Markdown
Contributor

Summary

Adds a schema-compare command to the test harness (tools/test-harness/) that compares static (Go tree-sitter) vs runtime (Python) schema generation for regression detection.

How it works

For each model, schema-compare:

  1. Builds with COG_STATIC_SCHEMA=1 → extracts OpenAPI schema from Docker label
  2. Builds without it → extracts OpenAPI schema from Docker label
  3. Compares the two JSON schemas for exact equality
  4. Reports a structured diff on mismatch showing the exact JSON paths that diverge
cd tools/test-harness

# Compare all non-GPU models
python -m harness schema-compare --no-gpu --cog-binary /path/to/cog

# Compare a specific fixture
python -m harness schema-compare --model fixture-scalar-types --cog-binary /path/to/cog

Local fixture models

7 local fixture models (fixtures/models/) covering the full input type matrix:

Fixture Coverage
scalar-types str, int, float, bool, Secret
optional-types PEP 604 X | None and Optional[X] for all types
list-types list[X] and List[X] for str, int, Path, File
optional-list-types list[X] | None and Optional[List[X]]
constraints-and-choices ge/le constraints, string/int choices
file-path-types Path, File, optional Path/File
complex-output BaseModel structured output

These use repo: local in the manifest and are loaded from fixtures/models/ without cloning.

Example output

Schema Comparison Report (CLI custom)
=====================================

  x fixture-scalar-types      schemas differ
      $.paths./predictions/{prediction_id}/cancel.post.parameters[0].schema.title: value mismatch
        static:  "Prediction_id"
        runtime: "Prediction Id"

No normalization is applied yet — the comparison is strict equality so we can see all differences before deciding which to fix vs which to accept.

@markphelps markphelps force-pushed the schema-compare-harness branch 3 times, most recently from 2c85d07 to a3e6902 Compare March 30, 2026 17:42
Add a new `schema-compare` command that builds each model twice — once
with COG_STATIC_SCHEMA=1 (Go tree-sitter) and once without (Python
runtime) — then compares the resulting OpenAPI schemas for exact JSON
equality. Differences are reported with a structured diff showing the
exact JSON paths that diverge.

Also add 7 local fixture models covering the full input type matrix:
scalar types, optional types (PEP 604 + typing.Optional), list types,
optional list types, constraints/choices, File/Path types, and
structured BaseModel output.
@markphelps markphelps force-pushed the schema-compare-harness branch from a3e6902 to 1ce7b49 Compare March 30, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant