Implement End-to-End Testing

[loispostula]

Problem

No E2E tests to validate that new versions won't break CI:

• Can't test full workflows before deployment
• Regressions only discovered in production CI
• No confidence when making changes
• Manual testing is time-consuming and incomplete

Proposed E2E Testing Strategy

Test Fixture Strategy (Hybrid Approach)

Use synthetic fixtures for most tests (predictable, fast) and one real-world repo for smoke tests (confidence).

test_data/e2e_fixtures/
├── simple_monorepo/          # Basic case
├── complex_monorepo/         # Real-world complexity
└── regression_tests/         # Known issues
    └── issue_001_ref_mismatch/

E2E Test Categories

Workflow Tests

• Test full check-workspace → publish workflow
• Test test execution workflow

Change Detection Tests

• Test transitive dependency detection
• Test cross-workspace dependencies
• Test whitelist/blacklist filtering

Publishing Tests

• Test GitHub release publishing
• Test Cargo registry publishing
• Test Docker image publishing
• Test NPM publishing
• Test S3 publishing

Parallel Execution Tests

• Test job_limit enforcement
• Test inner_job_limit enforcement
• Verify parallelism achieves expected speedup

Regression Tests

• Test for issue Release generate #1 (ref/SHA mismatch)
• Tests for any known historical bugs

Test Infrastructure

Test Harness

pub struct TestMonorepo {
    temp_dir: TempDir,
    git_repo: Repository,
}

pub struct MockRegistry { /* ... */ }
pub struct MockGithubServer { /* ... */ }

Test Utilities

• Helper functions to run commands
• JSON output parsing
• Result verification helpers

CI Integration

• Run E2E tests in CI on every PR
• Pre-release validation before creating releases
• Generate E2E test reports

Performance Benchmarks

Track performance over time to detect regressions:

• check-workspace on large repos
• parallel test execution
• publish workflow timing

Implementation Plan

Week 1: Foundation

• Create test harness (TestMonorepo, MockRegistry, etc.)
• Create simple test fixtures
• Set up E2E test directory structure

Week 2: Core Workflow Tests

• Test check-workspace E2E
• Test publish E2E with mocked services
• Test test command E2E

Week 3: Advanced Scenarios

• Test cross-workspace dependencies
• Test parallel execution
• Test error handling and recovery

Week 4: Regression Tests

• Add test for issue Release generate #1 (ref/SHA mismatch)
• Add tests for any known historical bugs
• Add performance benchmarks

Week 5: Real-World Testing

• Integrate real-world test repository
• Smoke test against real repo
• Document how to run E2E tests

Week 6: CI Integration

• Add E2E tests to CI pipeline
• Set up pre-release validation
• Generate test reports

Success Metrics

• E2E tests cover all major workflows (check, test, publish)
• E2E tests can run in < 5 minutes
• E2E tests catch the ref/SHA mismatch issue
• E2E tests run in CI on every PR
• Pre-release validation prevents broken releases
• At least one real-world repository tested

Test Coverage Goals

• Check workspace: 100% of scenarios
• Publish: 90%
• Tests: 90%
• Other commands: 80%

Labels

testing, priority:high