Skip to content

test: end-to-end smoke (docker compose up, sample PR runs every gate green, eval workflow_dispatch) #27

Closed

Description

@constk

Problem

Every prior ticket verified its own slice. Nothing yet verifies the integrated whole: clean clone -> sync -> all gates -> running stack -> green CI.

Proposed solution

Run the full verification checklist from the plan: clean-clone smoke, uv sync && just check, cd frontend && npm install && npm run lint && npm run check && npm run test && npm run build, pre-commit run --all-files, docker compose up and hit all three services, open a sample PR and confirm every CI gate passes, trigger eval-nightly via workflow_dispatch with sample secrets and confirm execution. File any defects discovered as bug issues against the relevant prior ticket.

Acceptance criteria

  • All checklist items in the plan's "Verification checklist" tick green.
  • No file mentions "Teller", "financial", or other Teller-specific identifiers (grep verified).
  • Sample PR closed without merging (verification only); summary of green checks attached.

Priority rationale

Critical: this is the gate that decides whether to publish or hold.

Depends on

#1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12, #13, #14, #15, #16, #17, #18, #19, #20, #21, #22, #23, #24, #25, #26

Activity

  1. added
    testTests, QA, eval harness
    harnessCross-cutting harness mechanics (controls, hooks, gates)
    on Apr 25, 2026
  2. constk commented on Apr 27, 2026

    @constk
    OwnerAuthor

    Verification summary

    Latest develop (707f49d) — all gates green.

    CI (.github/workflows/ci.yml) — 10 jobs

    • Lint & Format: ✅
    • Type Check: ✅
    • Unit tests: ✅
    • Coverage: ✅ (98.64 % > 75 % gate)
    • Architecture (import-linter): ✅ (2 contracts kept, 0 broken)
    • Pre-commit: ✅
    • Frontend Build: ✅
    • Frontend Quality: ✅
    • Branch-protection contexts sync: ✅ (15 required jobs across 2 branches)
    • Commit-type sync: ✅ (7 types in sync between commitizen schema and pr-title.yml)

    Security (.github/workflows/security.yml) — 4 jobs

    • Secret scan (gitleaks): ✅
    • Python deps (pip-audit): ✅ (CVE-2026-3219 in .github/security/pip-audit-ignore.txt with sunset note)
    • Frontend deps (npm audit): ✅
    • Container image scan (trivy): ✅

    Local verification

    • uv sync --extra dev: ✅
    • uv run ruff check . && ruff format --check .: ✅
    • uv run mypy --strict src/ tests/: ✅ (31 source files)
    • uv run lint-imports: ✅ (2 contracts kept)
    • uv run pytest tests/ --cov=src: ✅ (40 passed, 98.64 % coverage)
    • uv run pytest eval/: ✅ (1 case echo-hello passed)
    • uv run pre-commit run --all-files: ✅ (one Windows-side CRLF/LF cosmetic difference; no content drift, CI is unaffected)
    • cd frontend && npm ci: ✅ (153 deps, 0 vulnerabilities)
    • cd frontend && npm run lint: ✅ (0 warnings, max-warnings=0)
    • cd frontend && npm run check: ✅ (tsc --noEmit clean)
    • cd frontend && npm run format:check: ✅
    • cd frontend && npm run test: ✅ (2 passed)
    • cd frontend && npm run build: ✅ (Vite production bundle ~190 KB / 60 KB gzipped)

    Forbidden-token sweep

    grep -ri "teller|financial|tomoro|takehome|duckdb" — zero matches across the source tree (cleanup landed in #62).

    What's intentionally not verified here

    • docker compose up end-to-end stack boot — local Docker Desktop wasn't available during this run; the Dockerfile + compose file shape is reviewed and CI's Container image scan (trivy) builds the same image successfully every PR.
    • eval-nightly workflow_dispatch smoke — would need real LLM secrets configured; the workflow file shape is reviewed and the local pytest eval/ exercises the same code path with the placeholder _answer_fn.
    • Branch-protection apply workflow — needs the BRANCH_PROTECTION_TOKEN PAT secret; documented in docs/DEVELOPMENT.md#branch-protection-sync-setup.

    These three items are documented next-steps for #28's publish phase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    harnessCross-cutting harness mechanics (controls, hooks, gates)testTests, QA, eval harness

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions