Skip to content

docs: refresh README to reflect MEP v2 quality & safety features#13

Open
amar-python wants to merge 1 commit into
feature/schema-validationfrom
docs/readme-refresh
Open

docs: refresh README to reflect MEP v2 quality & safety features#13
amar-python wants to merge 1 commit into
feature/schema-validationfrom
docs/readme-refresh

Conversation

@amar-python

Copy link
Copy Markdown
Owner

Summary

Comprehensive README refresh documenting the current project state, including the code-quality work delivered in #10, #11, and #12.

Base branch / merge order: targets feature/schema-validation (like #10#12) because the backend/ layout only exists there. Since this README documents features introduced in #10#12, merge those three PRs first, then this one — otherwise a few referenced files (tools/verify_workflow_paths.py, docs/security/Rationale.md, .github/pull_request_template.md, backend/services/csv_parser.py) won't exist yet.

What changed

New: "MEP Backend — Reliability & Safety Features" section

  • Migration run lifecycle state machine — diagram + state list (CREATED → UPLOADING → VALIDATING → READY → MIGRATING → COMPLETED, with ERROR/FAILED paths), verified against ALLOWED_TRANSITIONS in backend/database/models.py on the fix(data): robust CSV parser, migration run state machine, shell script hardening #11 branch
  • CsvParser service — zero-byte/header-only rejection, ragged-row detection, UTF-8→latin-1 encoding fallback, suspicious-header flagging
  • Production safety gateALLOW_SCHEMA_AUTO_CREATE + fail-fast validation when APP_ENV=production
  • Typed API responses — Pydantic response_model on every endpoint; internal paths never leaked
  • Shell script hardening — table-name regex + 63-char limit, --env validation, eval removal
  • CI workflow path verificationtools/verify_workflow_paths.py usage
  • Security suppression policy — links docs/security/Rationale.md
  • Test instructions — how to run the backend suite, integration marker, legacy engine tests; notes growth from 38 → 100+ backend tests

Accuracy fixes to existing content

  • All legacy engine commands updated from stale root paths to backend/migration/... (setup/deploy/test runners, csv_loader.sh, csv_utilise.sh, evals/runner.py, PowerShell runners, psql invocation) — matches the actual tree and the CI workflow's own path
  • Engine "Repository Structure" tree re-rooted at backend/migration/ with an explanatory note; evals badge link fixed
  • Repository layout diagram updated (tools/, backend/tests/, docs/security|ci, PR template)

Contributing

  • Rewritten around the new PR template: Tests Added + Verified Workflow Paths checklists, response_model requirement, RunStatus lifecycle rule, suppression-documentation rule, correct test paths

Removed

  • Stale "Acknowledgements" section — ~140 lines of leftover session prompts referencing an unrelated repository (amar-python/TestUploadtoGIT)

Verification

  • Every backend/migration/... path referenced in the README was existence-checked against the working tree (all present)
  • State machine diagram cross-checked against ALLOWED_TRANSITIONS on fix/data-integrity-safety
  • Makefile csv-demo / csv-load targets confirmed to exist

Checklist

  • Tests Added — docs-only change; no code touched (existing suites unaffected)
  • Verified Workflow Paths — no workflow-referenced files moved

⚠️ Please review before merging — do not auto-merge. Recommended merge order: #7#10#11#12 → this PR.

- New 'MEP Backend — Reliability & Safety Features' section documenting:
  migration run lifecycle state machine (UPLOADING/VALIDATING/READY/ERROR),
  CsvParser service, production safety gate (ALLOW_SCHEMA_AUTO_CREATE),
  typed Pydantic response models, shell script hardening, CI workflow path
  verification (tools/verify_workflow_paths.py), and the security
  suppression policy (docs/security/Rationale.md)
- Repository layout updated (tools/, backend/tests/, docs/security|ci, PR template)
- 'Running the MEP backend tests' instructions; note suite growth 38 → 100+
- Fixed all stale legacy engine paths: build/, tests/, evals/ commands now
  use backend/migration/... (loader, utilise, deploy, test runners, evals)
- Engine repository-structure tree re-rooted at backend/migration/
- Contributing section rewritten around the new PR template with
  'Tests Added' and 'Verified Workflow Paths' checklists
- Removed stale 'Acknowledgements' section (leftover session prompts for an
  unrelated repository)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant