fix(data): robust CSV parser, migration run state machine, shell script hardening#11
Open
amar-python wants to merge 1 commit into
Open
fix(data): robust CSV parser, migration run state machine, shell script hardening#11amar-python wants to merge 1 commit into
amar-python wants to merge 1 commit into
Conversation
…shell hardening - Extract CSV parsing into services/csv_parser.py (CsvParser class) with explicit handling of zero-byte files, ragged rows, encoding fallback (UTF-8 BOM -> Latin-1 with warning) and SQL-injection-style headers; migration_service now delegates to it and logs every issue - Add READY and ERROR states to RunStatus with an explicit allowed- transition map, can_transition() and MigrationRun.transition_to() raising InvalidStateTransition on illegal moves - Flag runs as ERROR when an upload fails mid-way (with recovery path); validation now ends in READY (passed) or ERROR (failed) - Harden csv_loader.sh: strict table-name regex (blocks ';', '--', quotes, spaces) + 63-char limit; validate --env token in both csv_loader.sh and csv_utilise.sh; replace eval with indirect expansion in csv_utilise.sh - Frontend: status chip colors for new 'ready'/'error' states - Tests: test_csv_parser.py (28 cases) + test_run_lifecycle.py (39 cases)
This was referenced Jul 16, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Data Integrity & Safety (High Priority)
Based on
feature/schema-validation(PR #7) since that branch contains the restructuredbackend/layout these fixes target.4. Robust CSV Parser
backend/services/csv_parser.py— dedicatedCsvParserclass extracted frommigration_service._parse_csv_metadata(which previously swallowed every exception with a bareexcept: pass). It explicitly handles:empty_fileerror)encoding_fallbackwarning;,--, quotes, backslashes, comment markers, and standalone SQL keywords (DROP,DELETE, …) are flagged assuspicious_headererrors; legitimate headers likedrop_zone_idstill passmigration_servicenow delegates to it and logs every parse issue instead of silently ignoring it.backend/tests/test_csv_parser.py— 28 unit tests covering all four failure classes plus happy paths.5. Migration Lifecycle State Machine
RunStatusgains explicitREADYandERRORstates (joining CREATED / UPLOADING / VALIDATING / MIGRATING / COMPLETED / FAILED).ALLOWED_TRANSITIONSmap +can_transition()+MigrationRun.transition_to(), which raisesInvalidStateTransitionon illegal moves (e.g. COMPLETED → UPLOADING).ERROR(previously it lingered in UPLOADING); the run recovers by re-uploading.READY(passed) orERROR(failed) — previously it always reverted to CREATED, even on failure (RunStatus.CREATED if passed else RunStatus.CREATEDbug).migration_service,schema_service,execution_service) route status changes throughtransition_to().backend/tests/test_run_lifecycle.py— transition-rule matrix,transition_toguard tests, and API-level tests including a simulated mid-upload disk failure → ERROR → recovery.6. Shell Script Hardening (
backend/migration/build/)csv_loader.sh: strict table-name validator after sanitisation —^[a-zA-Z_][a-zA-Z0-9_]*$(blocks;,--, quotes, spaces) plus PostgreSQL's 63-char identifier limit;--envtoken validated before use.csv_utilise.sh:--envtoken validated;eval echo "\$PG_DB_${E}"replaced with safe bash indirect expansion (${!var}) — removes a shell-injection path. (Its table-name validator already existed and is verified by tests.)shellcheckwith no word-splitting (SC2086) findings; all variable expansions double-quoted.Verification
pytest backend/tests/→ 108 passed (38 existing + 70 new)pytest -m "unit or regression or security or snapshot" backend/migration/tests/→ 39 passed (no regressions in legacy loader/utilise tests)csv_loader.sh '/tmp/bad;drop.csv'→ rejected;--env 'dev; rm -rf /tmp/x'→ rejected by both scripts; valid config still resolves DB/schema via indirect expansionbash -n+ shellcheck clean; flake8 clean on all new PythonChecklist