Skip to content

LIMITATIONS.md

Latest commit

 

History

History
296 lines (204 loc) · 8.78 KB

LIMITATIONS.md

File metadata and controls

296 lines (204 loc) · 8.78 KB

CLI Linter Limitations

ConfigHub CLI Reference: For authoritative command documentation, use CONFIGHUB_AGENT=1 cub --help-overview or see docs.confighub.com. This document describes linter limitations.

This document honestly describes what the ConfigHub CLI linter cannot do. Understanding these limitations is critical for using the tool appropriately.

Philosophy

This tool is a linter that provides quick feedback on common errors. It is not a comprehensive validator and cannot replace testing with the actual cub CLI.

Think of it like:

  • ESLint for JavaScript (catches common mistakes, not all errors)
  • shellcheck for bash (helpful hints, not execution validation)
  • NOT like: A compiler (which guarantees correctness)

Known Limitations

1. Out of Date with CLI

Last synchronized: 2025-10-12 (cub commit a9e49ba)

Problem: The linter has hardcoded command lists that may not match your cub CLI version.

Missing entities (as of last check):

  • helm - Helm chart management
  • trigger - Trigger commands
  • invocation - Function invocation tracking
  • mutation - Mutation management
  • organization / organization-member - Org management
  • tag - Tag entity commands
  • unit-action / unit-event - Unit lifecycle
  • user - User management
  • view - View commands

Missing unit verbs (as of last check):

  • approve - Unit approval workflow
  • edit - Edit in system editor
  • import - Import from various sources
  • refresh - Refresh from target
  • tag - Tag management

Wrong verb names:

  • Linter has get-live-state, actual CLI has livestate

Impact: Valid commands may be rejected (false positives) or invalid commands may pass (false negatives).

Workaround: Always test with cub CLI. Run cub [entity] --help to see actual available commands.

2. No Version Detection

Problem: Linter doesn't check your cub version or adjust rules accordingly.

Scenario:

  • You upgrade cub to latest version
  • New commands are added
  • Linter rejects them as invalid

Impact: Linter will lag behind CLI evolution.

Workaround: Check cub version and compare with "Last synchronized" date above. If significantly different, linter may be outdated.

3. Missing New Flags

Known missing flags:

  • --where-data - Query actual config data (example: --where-data "spec.replicas > 3")
  • --resource-type - Filter by Kubernetes resource type
  • Possibly others added since last sync

Impact: Commands using these flags may be incorrectly flagged as invalid.

Workaround: Consult cub [entity] [verb] --help for actual flag list.

4. Cannot Validate Field Names

Problem: Linter doesn't know which fields exist for each entity.

Examples linter CANNOT catch:

# Typo in label key - linter says VALID
cub unit list --where "Labels.enviornment = 'prod'"
#                             ^ typo

# Nonexistent field - linter says VALID
cub unit list --where "Labels.nonexistent = 'value'"

# Wrong field type - linter says VALID
cub unit list --where "CreatedAt = 12345"  # Should be timestamp string

Impact: Field-level errors slip through.

Workaround: Test with actual CLI. Review error messages carefully.

5. Cannot Validate Complex WHERE Clauses

Problem: WHERE clause validation uses regex, not proper parsing.

Examples that may fail:

# Nested parentheses
--where "(A = 1 AND B = 2) OR (C = 3 AND D = 4)"  # OR not supported anyway

# Complex string escaping
--where "Labels.name = 'value with \"quotes\"'"

# Advanced operators
--where "LEN(ApprovedBy) > 0 AND ApprovedBy ? 'uuid'"  # Array operations

Impact: May incorrectly reject valid complex clauses or accept invalid ones.

Workaround: Test complex WHERE clauses with actual CLI first.

6. Cannot Validate Command Sequences

Problem: Linter validates commands in isolation.

Examples linter CANNOT catch:

# Try to apply before creating
cub unit apply myunit --space dev  # Valid syntax
cub unit create myunit --space dev myunit.yaml  # Valid syntax
# But wrong order! Should create before apply

# Approve without proper permission
cub unit approve myunit --space prod  # Syntax valid
# But user might not have approval permissions

# Apply to non-existent space
cub unit apply myunit --space nonexistent  # Syntax valid
# But space doesn't exist

Impact: Workflow errors not caught.

Workaround: Test full workflows end-to-end.

7. Cannot Validate Against ConfigHub State

Problem: Linter is stateless and doesn't contact ConfigHub API.

Examples linter CANNOT catch:

# Unit already exists
cub unit create existing-unit ...  # Will fail at runtime

# Space doesn't exist
cub unit list --space nonexistent  # Will fail at runtime

# Circular upstream references
cub unit create unit-a --upstream-unit unit-b
cub unit create unit-b --upstream-unit unit-a  # Cycle!

# Permission denied
cub unit delete protected-unit --space prod  # User lacks permission

Impact: Runtime errors not predicted.

Workaround: Use actual CLI with real ConfigHub connection.

8. Hardcoded Validation Rules

Problem: Rules are static code, not extracted from CLI.

Architecture:

# Line 143 in cub-test-framework.sh
local valid_entities="space unit filter ..."  # HARDCODED

# Lines 150-176
case "$entity" in
    space)
        local valid_verbs="create get list ..."  # HARDCODED

Impact: Every CLI update requires manual code changes.

ConfigHub maintainer (Brian Grant) feedback:

"This is looking at command syntax rather than executing the commands? That may be difficult to keep up to date"

Workaround: Contribute updates when you notice missing commands, or use dynamic help parsing (future enhancement).

9. False Confidence from Test Pass Rate

Test results: 39/39 tests passing (100%)

But:

  • Tests validate KNOWN commands only
  • Don't test NEW CLI features
  • Don't test version compatibility
  • Don't test field typos
  • Tests are also hardcoded

100% pass rate means: "Our hardcoded linter matches our hardcoded tests"

NOT: "Linter catches all errors"

Impact: May give false sense of security.

Workaround: Don't rely solely on linter. Always run actual CLI tests.

10. No Dynamic Help Parsing

Brian Grant's recommendation:

"It should set CONFIGHUB_AGENT=1 and do cub --help-overview, and also do --help for every command it wants to use."

We don't do this (yet):

  • ❌ No CONFIGHUB_AGENT=1 environment variable
  • ❌ No parsing of cub --help output
  • ❌ No extraction of commands from help text
  • ❌ Rules don't adapt to installed CLI version

Impact: Linter can't adapt to CLI changes automatically.

Workaround: Future enhancement planned (see Phase 2 of improvement plan).

What the Linter IS Good For

Despite limitations, the linter is useful for:

✅ Catching obvious typos - Entity/verb name mistakes ✅ Flag combination errors - Missing required flags ✅ Quick feedback - Faster than running actual CLI ✅ Common patterns - Based on maintainer feedback ✅ WHERE clause basics - Simple syntax errors ✅ CI integration - Quick sanity check before full tests ✅ Learning tool - Shows common error patterns

What You Should Do Instead

For reliable validation:

  1. Use actual CLI:

    cub unit create test --space dev test.yaml
# Real error messages
# Real validation
# Real state checking

  2. Read CLI help:

    export CONFIGHUB_AGENT=1
cub --help-overview
cub unit --help
cub unit create --help

  3. Run integration tests:

    • Deploy to test environment
    • Verify actual behavior
    • Check ConfigHub state

  4. Use linter as first line:

    • Catch obvious errors quickly
    • Then test with actual CLI
    • Don't stop at linter validation

Maintenance Strategy

How we'll keep this updated:

  1. Quarterly sync with ConfigHub CLI (planned)
  2. Community contributions when gaps found
  3. CI against ConfigHub HEAD (planned Phase 4)
  4. Dynamic help parsing (planned Phase 2)

How you can help:

  • Report missing commands via GitHub issues
  • Submit PRs to update entity/verb lists
  • Share examples of false positives/negatives

Version History

Date CLI Version Linter Changes
2025-10-12 a9e49ba Initial hardcoded rules
(future) TBD Add dynamic help parsing

Conclusion

Use this tool wisely:

  • ✅ As a linter (quick hints)
  • ✅ For catching common mistakes
  • ✅ As part of broader testing strategy

Don't use as:

  • ❌ Comprehensive validator
  • ❌ Substitute for CLI testing
  • ❌ Final authority on command validity

Always remember: The only source of truth is the cub CLI itself.

When in doubt: cub [entity] [verb] --help