Skip to content

Add config-driven CLI syntax validator for downstream drift detection #10

@iamh2o

Description

@iamh2o

Summary

cli-core-yo already enforces parts of the v2 CLI contract at app construction and registration time, and it exposes cli_core_yo.conformance helpers for downstream pytest suites. What is still missing is a reusable, config-driven CLI syntax validator that related CLIs can run to detect and report drift from shared core operations.

We need a validator that downstream CLIs can use to enforce or complain about drift from a declared validation config instead of relying on ad hoc repo-local assertions.

Problem

  • Related CLIs can drift from core root behavior and command structure even when they embed cli-core-yo.
  • Current enforcement is split across registry validation, app bootstrap rules, and downstream tests.
  • cli_core_yo.conformance helps write tests, but it does not provide a normalized command-tree validator or a declarative drift report.
  • We need both strict enforcement for CI and advisory reporting for migration / rollout work.

Proposed feature

Add a reusable CLI syntax validation surface to cli-core-yo.

This can land as a new module such as cli_core_yo.validation or as an extension of cli_core_yo.conformance, but the core behavior should be:

  • accept a validation config that defines expected core operations and allowed deviations
  • inspect a downstream CLI and normalize its command tree / syntax surface
  • emit structured findings for drift, with both human-readable and machine-readable output
  • support both fail-on-drift mode and warn-only mode
  • be usable from downstream pytest suites, CI jobs, and optional local validation commands

Validation config should be able to express

  • required root-owned operations and global flags
  • required built-ins such as version, info, and optional shared groups when enabled
  • required or forbidden command paths
  • naming constraints for commands/groups
  • per-command policy expectations such as supports_json, supports_dry_run, runtime_guard, and possibly mutates_state
  • reserved-name violations or other framework-owned surface drift
  • allowlisted exceptions so downstream repos can intentionally suppress known deviations while they migrate

Implementation notes

  • Prefer validating a normalized app/registry model instead of scraping help text only.
  • Keep finding codes stable so downstream repos can gate on them in CI.
  • The validator should return a structured report object that downstream code can inspect programmatically.
  • A lightweight config format is fine, but it should be straightforward for related CLIs to share and version.

Acceptance criteria

  • A downstream CLI can run the validator against its command surface using a declared validation config.
  • The validator can distinguish pass / warn / fail results.
  • Findings identify the command path or root surface that drifted.
  • There is a machine-readable report format suitable for CI gating.
  • There are tests covering at least:
    • missing required built-ins
    • unexpected or forbidden command paths
    • invalid command naming / syntax
    • root flag drift
    • command policy mismatch
    • allowlisted drift behavior
  • Documentation shows how a related CLI wires this into tests or validation workflows.

Why this belongs here

cli-core-yo is already the shared enforcement layer for downstream CLI behavior. A syntax validator closes the remaining gap between framework-owned contract rules and downstream repo compliance by making drift detection reusable, explicit, and automatable.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions