ci: add Linux quality gate; docs: setup runbook#2
Merged
Conversation
CI previously ran only a Windows unittest job, leaving most of the repo's own quality tooling unused on every merge: the pytest marker tiers and coverage (make test-free), flake8+bandit (make lint), the structural health check, the 23 offline Tier P eval scenarios, and every PG-gated suite (integration/e2e/parity pytest tiers, eval Tiers I and S with the 85-assertion SQL suite) which skipped forever without a database. New .github/workflows/quality-gate.yml adds two ubuntu jobs: - free-tier: make test-free + make lint + make health + evals Tier P. No database; fast merge gate. - integration-postgres: postgres:16-alpine service container, CI-only build/config.local.env, then evals Tiers P/I/S (Tier I deploys env_dev.sql twice as the idempotency check; Tier S runs the SQL suite), deploys env_test.sql, and runs the e2e/integration/parity pytest tiers that were previously always skipped. Making lint blocking surfaced minor issues, fixed here: E302/E305/E231 in build/csv/validator.py, and a bandit B608 on a query built from the hardcoded _DEV_SEED_TABLES constant (annotated nosec). E221/E272 join the flake8 ignore list — aligned assignments are this repo's idiom. Also refreshes ARCHITECTURE.md's tests/ table, which listed 2 of the 9 test modules. Unblocks (not implemented here): BR-15 connection-limit assertions, BR-01 cross-env parity tier, BR-02 engine coverage — all previously blocked on having PG in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBrxqChRJtxdvSpFhiUWUy
Phase-by-phase commands (prereqs, clone, preflight, configure, deploy, verify, CSV operations), each labelled with the terminal it must run in (Git Bash vs PowerShell) — the repo's .sh scripts fail in PowerShell/cmd and that mismatch is the most common setup failure. Ends with a four-step troubleshooting order for DB-dependent failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBrxqChRJtxdvSpFhiUWUy
The env_*.sql deploy scripts intentionally do not create their own databases — deploy_all.sh does that at the shell level first (CREATE DATABASE cannot run inside te_core_schema.sql's transaction blocks; see build/te_core_schema.sql:41). The integration job invoked env_dev.sql via the eval runner without that precursor, so on the virgin CI Postgres every Tier I/S scenario failed with 'database "te_mgmt_dev" does not exist'. Mirror deploy_all.sh's idempotent create for te_mgmt_dev and te_mgmt_test before the eval step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBrxqChRJtxdvSpFhiUWUy
Verified end-to-end against a live PostgreSQL 16: 142/142 SQL assertions,
25/25 eval scenarios (Tiers P/I/S), and the e2e/integration/parity pytest
tiers all pass with these changes.
Four independent breakages, all in the path the new CI integration job
exercises for the first time:
1. tests/suites/test_02/03/04 still used :"var" psql substitution inside
DO $$ blocks, where psql never substitutes — the raw ':' reached the
server as a syntax error. Converted to the same pattern the already-
repaired test_01/test_05 use: bare table names (search_path is set by
run_all_tests.sql) and current_setting('te.schema_name') where the
schema name is needed as a value.
2. test_05's R01/R02 trigger assertions spliced raw unquoted timestamps
into the assert_true expression ("SELECT ( 2026-07-14 ... > ...)"),
a syntax error whenever they ran. The comparison is now evaluated in
plpgsql and passed as a boolean literal.
3. evals/runner.py's Tier S invocation never passed the app_user and
conn_limit psql vars that run_all_tests.sql now requires for its
set_config() preamble, so the suite aborted before producing output.
4. runner.py's summary parser only understood a whitespace-separated
summary row; the suite emits a psql table row with pipes. Parser now
strips pipes and finds the percentage token positionally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBrxqChRJtxdvSpFhiUWUy
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI previously ran only a Windows unittest job, leaving most of the repo's own quality tooling unused on every merge. This PR adds a Linux quality gate and a setup runbook.
Commit 1 — ci: add Linux quality gate with postgres integration job
New
.github/workflows/quality-gate.ymlwith two ubuntu jobs:make test-free(pytest marker tiers) +make lint(flake8 + bandit, now blocking) +make health+ evals Tier P (23 offline scenarios). No database; fast merge gate.postgres:16-alpineservice container, CI-onlybuild/config.local.env, then evals Tiers P/I/S (Tier I deploysenv_dev.sqltwice as the idempotency check; Tier S runs the SQL assertion suite), deploysenv_test.sql, and runs the previously always-skippede2e/integration/paritypytest tiers.Making lint blocking surfaced minor issues, fixed here: E302/E305/E231 in
build/csv/validator.py, a bandit B608 on a query built from a hardcoded constant (annotatednosec), and E221/E272 added to the flake8 ignore list (aligned assignments are this repo's idiom). Also refreshesARCHITECTURE.md's tests/ table (2 → 10 rows).Commit 2 — docs: add SETUP_RUNBOOK.md
Phase-by-phase commands for standing up a fresh environment, each labelled with the terminal it must run in (Git Bash vs PowerShell), plus a troubleshooting order for DB-dependent failures.
Test plan
make test-free— 39 passed locally on this branchmake lint— flake8 + bandit cleanmake health— 85 checks passevals/runner.py --tiers p— 23/23 offline scenarios passintegration-postgresjob — first-ever execution happens on this PR's CI run