Skip to content

README.md

Latest commit

 

History

History
177 lines (128 loc) · 11 KB

README.md

File metadata and controls

177 lines (128 loc) · 11 KB
Raw

compat/ — cross-CLI conformance library

This module is the machine-attested half of CONTRACT.md. Every *-export-cli is expected to wire it into CI so the contract's "✓" status table stops being purely human-attested.

It is a Go module (github.com/quantcli/common/compat) under the compat/ subdirectory of quantcli/common. Stdlib only.

Design at a glance

  • Black-box. The library never imports a CLI's internal packages. It shells out to the binary and asserts on stdout, stderr, and exit code.
  • One subpackage per contract section. dates/ (CONTRACT §2–§3) and formats/ (CONTRACT §4) ship today. auth/ and prime/ are expected to follow. The convention is compat/<section> where <section> is the CONTRACT.md section being attested; each subpackage exposes a single RunContract(t, runner) entry point.
  • Exporters consume via a build-tagged _test.go. No production import; compat tests do not ship in the released binary.
  • Hermetic by default. compat.Runner runs the binary with an empty environment unless callers opt into specific variables. Subtests that need to assert "no network call" set proxy env vars to an unreachable address.

Usage from an exporter

Add a single file in your repo (e.g. compat_test.go) gated behind a build tag so it does not run as part of the default go test ./...:

//go:build compat

package main_test

import (
    "os"
    "testing"

    "github.com/quantcli/common/compat"
    "github.com/quantcli/common/compat/dates"
    "github.com/quantcli/common/compat/formats"
)

func TestContractDates(t *testing.T) {
    bin := os.Getenv("EXPORT_CLI_BIN")
    if bin == "" {
        t.Skip("EXPORT_CLI_BIN not set; skipping compat suite")
    }
    dates.RunContract(t, compat.Runner{Binary: bin})
}

func TestContractFormats(t *testing.T) {
    bin := os.Getenv("EXPORT_CLI_BIN")
    if bin == "" {
        t.Skip("EXPORT_CLI_BIN not set; skipping compat suite")
    }
    formats.RunContract(t, compat.Runner{Binary: bin})
}

Cobra-based CLIs: date flags on subcommands

When --since/--until live on subcommands (the crono / liftoff / withings pattern), set Subcommands and the suite will dispatch per-subcommand:

func TestContractDates(t *testing.T) {
    bin := os.Getenv("EXPORT_CLI_BIN")
    if bin == "" {
        t.Skip("EXPORT_CLI_BIN not set; skipping compat suite")
    }
    dates.RunContract(t, compat.Runner{
        Binary: bin,
        Subcommands: []string{
            "biometrics", "exercises", "nutrition", "servings", "notes",
        },
    })
}

Each subcommand is verified under a subcommand=NAME/... subtree, so a regression in any single one fails as a named subtest instead of masking the rest.

Nested-path subcommands are supported by passing a space-separated string. The dispatcher splits on whitespace so each segment becomes its own argv entry, which is what cobra's command tree resolves against:

dates.RunContract(t, compat.Runner{
    Binary: bin,
    Subcommands: []string{
        "bodyweights list", "bodyweights stats",
        "workouts list", "workouts stats",
    },
})

This is the pattern liftoff-export uses, where --since/--until live on two-level leaves (liftoff-export workouts stats) rather than top-level subcommands.

CI workflow

- name: build
  run: go build -o /tmp/cli .
- name: compat tests
  env:
    EXPORT_CLI_BIN: /tmp/cli
  run: go test -tags=compat ./...

The exporter does not need a separate go.mod for compat tests — the standard require github.com/quantcli/common/compat vX.Y.Z line in the exporter's existing go.mod is enough.

What dates.RunContract covers today

Subtest Contract What it asserts
HelpDocumentsDateFlags §3 --help mentions --since and --until and exits 0.
InvalidSinceValueFails §3, §4 --since obviously-not-a-date exits non-zero, writes to stderr, leaves stdout empty.
HelpIsHermetic §7 --help succeeds with all HTTP proxies pointed at an unreachable address.
FlagValidationIsHermetic §7 A parse failure also produces no successful outbound request.

When compat.Runner.Subcommands is set, every row above runs once per declared subcommand under subcommand=NAME/....

What formats.RunContract covers today

Subtest Contract What it asserts
HelpDocumentsFormatFlag §4 --help mentions --format and exits 0.
UnknownFormatFails §4 --format obviously-not-a-format exits non-zero, writes to stderr, leaves stdout empty.
FlagValidationIsHermetic §7 The unknown-format parse failure makes no successful outbound request.
JSONIsArray §4 --format json exits 0 and emits stdout that unmarshals as []any.
CSVHasHeader §4 --format csv exits 0 and emits at least one non-empty line on stdout (the header row, present even on an empty result).
DefaultIsMarkdown §4 No --format flag produces byte-identical stdout to --format markdown.

JSONIsArray, CSVHasHeader, and DefaultIsMarkdown invoke the data path with no extra args beyond --format. Integrators whose data path needs credentials or other env to succeed pass them via compat.Runner.Env; integrators whose data path is not runnable in CI at all today set compat.Runner.SkipDataPath: true (see below). As with dates, subcommand=NAME/... subtest groups fire when Subcommands is set.

Partial-codec exporters: Runner.SupportedFormats

§4 lists three required codecs (markdown, json, csv), but crono-export-cli and liftoff-export-cli reject --format csv today — only withings-export-cli implements all three. The bundle is still adoptable by partial-codec exporters via compat.Runner.SupportedFormats:

formats.RunContract(t, compat.Runner{
    Binary:           os.Getenv("EXPORT_CLI_BIN"),
    SupportedFormats: []string{"markdown", "json"}, // declare what you implement
})

Semantics:

  • Nil (default) — exporter declares the full §4 surface. Every subtest runs.
  • Non-nil subset — JSONIsArray, CSVHasHeader, and DefaultIsMarkdown skip via t.Skipf when their codec is not in the slice. The skip names the missing codec so the gap is visible in test output rather than masked. The parse-level subtests (HelpDocumentsFormatFlag, UnknownFormatFails, FlagValidationIsHermetic) run regardless because they assert on the --format flag itself, not on a specific codec.
  • Empty slice — declares zero codecs. Rarely useful; effectively disables the data-path subtests. Use nil to mean "all of §4".

Per-CLI flip plan. When a consumer wires the bundle into its CI with SupportedFormats matching its actual surface, the CONTRACT.md Status table's --format row for that CLI flips to machine for the codecs it declares. Codecs the CLI does not implement remain human-attested per-cell until the writer lands. Today the Status table has one cell per (CLI, codec) so the flip is per-cell, not per-row.

Credentialless CI: Runner.SkipDataPath

The bundle's three data-path subtests (JSONIsArray, CSVHasHeader, DefaultIsMarkdown) invoke the CLI with a real --format value and expect a clean exit — which means the data path has to be runnable in the CI environment. For crono (env-var auth), liftoff (stored OAuth token), and withings (OAuth refresh token), that is not true today: a vanilla --format json invocation under compat's default empty env exits non-zero with "not logged in" before the JSON-array check fires.

compat.Runner.SkipDataPath is the explicit escape hatch: when true, the data-path subtests skip with reason data-path subtests disabled via Runner.SkipDataPath. The parse-level subtests still run because they assert on flag parsing, not on a working data path. This lets an exporter wire the bundle in for parse-level attestation today, then flip the bool back to false later (once auth is mockable in CI or secrets are provisioned) to promote its codec rows from human to machine.

formats.RunContract(t, compat.Runner{
    Binary:       os.Getenv("EXPORT_CLI_BIN"),
    SkipDataPath: true, // CI does not have upstream credentials
})

SkipDataPath and SupportedFormats compose: nil SupportedFormats + SkipDataPath: true is the typical liftoff/crono shape today (declare the full surface, but skip the data-path subtests until creds are available). Use SupportedFormats: []string{...} instead of SkipDataPath: true when the CLI structurally lacks a codec — those are two different gaps and should not be confused.

What it does NOT cover yet

The actual local-midnight semantics of --since 2026-04-15 (the harmonization that just landed across crono/liftoff/withings) is still human-attested in the status table. Asserting it black-box requires either:

  • a --print-resolved-window-style affordance on every CLI (a substantive contract change, intentionally out of scope of the first compat-test cut), or
  • per-upstream recorded HTTP fixtures (heavy, and pinned to specific API shapes).

When that affordance lands, the test belongs here as dates.LocalMidnightSemantics.

Adding a new contract test

  1. Decide which CONTRACT.md section it pins down. If no subpackage exists for that section, create one (compat/<section>/).
  2. Write the assertion as a function that takes *testing.T and compat.Runner.
  3. Wire it into the section's RunContract. Subtests are t.Run-scoped so one failure does not mask the rest.
  4. Update this README's table and CONTRACT.md's status-attestation note in the same PR.

Self-test

This module has its own tests that run each bundle against a stub CLI in internal/stubcli/. The stub is intentionally narrow — it exists so go test ./... from this module's root proves the library compiles and the assertions fire correctly, without depending on any of the real export-CLIs. Failures in the self-test mean the library has a bug; failures in an exporter's compat test mean the exporter drifted from the contract.

The stub has two modes (STUBCLI_MODE=flat and STUBCLI_MODE=cobra). The flat-mode self-tests exercise the original Runner shape; the cobra-mode self-tests exercise Subcommands-based dispatch. In cobra mode, the stub's root --help deliberately omits --since/--until/--format, so the cobra-mode self-tests fail fast if compat.Runner ever stops prepending the subcommand. The stub emits an empty data set per --format codec ([] for json, a single header row for csv, nothing for markdown) so the formats bundle's data-path subtests run hermetically. There is also a focused unit test for Runner.WithSubcommand using an argecho helper that just prints os.Args.

The stub additionally honors STUBCLI_FORMATS (comma-separated) to restrict which codecs it will accept at parse time — used by the formats bundle's TestRunContract_PartialCodec_SkipsCSV self-test. Setting STUBCLI_FORMATS=markdown,json makes the stub reject --format csv, so the partial-codec self-test proves the SupportedFormats skip is actually load-bearing: without it, the test would fail at CSVHasHeader because the stub rejects csv.