feat: Migration Evaluation Platform (MEP) — Complete Build (Phases 0–8)#7
Open
amar-python wants to merge 11 commits into
Open
feat: Migration Evaluation Platform (MEP) — Complete Build (Phases 0–8)#7amar-python wants to merge 11 commits into
amar-python wants to merge 11 commits into
Conversation
- Restructured repo: existing engine preserved in backend/migration/ - FastAPI backend skeleton with health endpoint - React + TypeScript frontend shell with Azure Portal UI - Docker Compose stack (React + FastAPI + PostgreSQL) - GitHub Actions CI/CD pipeline - Documentation (Architecture, DeveloperGuide, API, ADRs)
…orkflows permission — move to .github/workflows/ci.yml via web UI)
- Database: MigrationRun and UploadedFile models (auto-created on startup) - API: CRUD endpoints for migration runs + multi-file upload/list/delete - Service: migration_service.py with CSV metadata parsing (row/column counts) - Frontend: New Migration page with 3-step wizard (Configure → Upload → Summary) - Drag-and-drop zone with multi-file support - Upload progress bar - File list table with size, rows, columns, delete action - Frontend: Migration Runs page with live data table - Tests: 17 new API tests (19 total) using SQLite test DB - All tests passing, frontend build successful
…aluation, reports, dashboard
Slice 1 — Schema Discovery + Validation (Phases 2 & 4):
- Schema inference: integer, decimal, date, boolean, text detection
- Nullability, uniqueness, sample values per column
- Validation: duplicate columns, empty headers, null checks, duplicate rows, mixed types
- POST /api/migrations/{id}/validate endpoint
- Validation page with expandable per-file schema table + issues
Slice 2 — Migration + Evaluation (Phases 5 & 6):
- Staging table creation with inferred PostgreSQL types
- Bulk CSV loading via parameterized INSERT
- Quality evaluation: row count match, null %, duplicate detection
- Quality score (0-100) with PASS/FAIL verdict
- POST /api/migrations/{id}/execute and /evaluate endpoints
- Execute/Evaluate actions on Migration Runs page with result dialogs
Slice 3 — Reports + Dashboard (Phases 7 & 8):
- JSON and HTML report generation with validation/evaluation data
- Self-contained styled HTML reports
- Report download endpoint
- Dashboard with stat cards, status breakdown, recent runs table
- GET /api/dashboard endpoint
Also:
- Shared test conftest.py (fixes cross-test DB conflicts)
- 38 backend tests passing (19 new)
- Frontend builds clean (630 KB)
- Updated API.md documentation
- History page with full chronological run log
- MigrationRuns page with execute/evaluate action buttons
…umentation suite - Administration page: database connection test, platform config table, security/RBAC toggles (planned), system info panel - docs/Database.md: full schema reference (migration_runs, uploaded_files, staging tables, ER diagram, Alembic migration path) - docs/TestingStrategy.md: test pyramid, 38-test inventory, validation check catalogue, evaluation scoring, CI integration, future plans - docs/UserGuide.md: end-to-end walkthrough of all 7 pages with tables and troubleshooting guide - docs/Deployment.md: Docker Compose, standalone, production checklist, Azure Container Apps instructions, health checks - docs/ADR/ADR-003-Upload-Workflow.md: 6-stage pipeline design, staging table rationale, quality score thresholds, alternatives considered - CI pipeline: ci.yml ready at .github/workflows/ci.yml and docs/ci/ (requires manual move via GitHub web UI due to App permission) - Frontend rebuild: 645 KB bundle, all pages functional, no placeholders
amar-python
force-pushed
the
feature/schema-validation
branch
from
July 15, 2026 10:07
990fee0 to
5e4980a
Compare
amar-python
changed the title
… backend/migration/ restructure The Phase 0 restructure moved the original engine into backend/migration/ but these files still referenced old root-level paths (tests/, build/, evals/). Updated: Makefile, pytest.ini, scripts/health_check.py, scripts/lint.sh, scripts/select_tests.py, backend/migration/tests/run_python_tests.ps1. Note: .github/workflows/*.yml also need the same path updates but cannot be pushed via the GitHub App (lacks workflows permission). The corrected versions are saved at docs/ci/ for manual application.
GitHub App lacks workflows permission to push .github/workflows/ changes. These corrected files update paths for the backend/migration/ restructure. Copy them to .github/workflows/ via the GitHub web UI to fix CI.
Changes made as "The argument 'backend/migration/tests/run_python_tests.ps1' to the -File parameter does not exist."
This was referenced Jul 16, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migration Evaluation Platform (MEP) — Complete Build
This PR contains the entire MEP application (Phases 0–8), rebased cleanly on main.
The CI workflows are failing because the Phase 0 restructure moved the original engine to
backend/migration/, but the workflow files still reference old root-level paths (tests/,build/,evals/).The fix is ready — corrected workflow files are at:
docs/ci/quality-gate.yml→ copy to.github/workflows/quality-gate.ymldocs/ci/python-validator-tests.yml→ copy to.github/workflows/python-validator-tests.ymlThe GitHub App lacks
workflowspermission, so these must be applied via the GitHub web UI (edit each file in-place) or pushed from your local machine. All non-workflow path fixes (Makefile, pytest.ini, scripts) are already included in this PR.Phase 0 — Foundation
backend/migration/Phase 1 — CSV Upload
MigrationRunandUploadedFileORM models with status lifecyclePhases 2 & 4 — Schema Discovery + Validation
Phases 5 & 6 — Migration Execution + Evaluation
Phases 7 & 8 — Reports + Dashboard
Additional Deliverables