Skip to content

Commit dc14eed

Browse files
mvalancymvalancy
committed
Add comprehensive test monitoring and timing analysis tools
- Create monitor-test-progress.sh for real-time test status updates - Shows process status, elapsed time, active tests - Auto-refreshing display with detailed breakdowns - Tracks Playwright, Vitest, npm test processes - Create analyze-test-timing.sh for post-run analysis - Parses log files for duration metrics - Estimates step durations from file timestamps - Provides intelligent optimization recommendations - Add timing infrastructure to test-vm-e2e.sh - start_step() and end_step() helper functions - Timing tracking with associative arrays - Console output showing step completion times - Add detailed VISUAL_REGRESSION_README.md - Complete documentation of visual regression suite - Device coverage list (21 configurations) - Integration guide for GraphDone-DevOps - Performance characteristics and best practices These tools address the need for better visibility into long-running E2E test execution and provide data for performance optimization.
1 parent 9c063fa commit dc14eed

5 files changed

Lines changed: 800 additions & 0 deletions

File tree

‎tests/e2e/VISUAL_REGRESSION_README.md‎

Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
# Visual Regression Testing Suite
2+
3+
## Overview
4+
5+
Comprehensive screenshot collection system for GraphDone UI monitoring and visual regression testing. This suite captures screenshots across 21 different device configurations and 10+ core application screens, generating 250-300 total screenshots per test run.
6+
7+
## Purpose
8+
9+
- **Visual Regression Testing**: Compare UI changes across releases
10+
- **DevOps Monitoring**: Automated visual change detection in CI/CD pipeline
11+
- **Cross-Device Compatibility**: Verify UI renders correctly on all devices
12+
- **UI/UX Documentation**: Maintain visual records of application state
13+
- **Design Review**: Provide stakeholders with visual references
14+
15+
## Device Coverage
16+
17+
### Mobile Phones (Portrait)
18+
- iPhone SE (375×667, 2x)
19+
- iPhone 12/13/14 (390×844, 3x)
20+
- iPhone 14 Pro Max (430×932, 3x)
21+
- Samsung Galaxy S21 (360×800, 3x)
22+
- Google Pixel 7 (412×915, 2.625x)
23+
24+
### Mobile Phones (Landscape)
25+
- iPhone 14 Landscape (844×390, 3x)
26+
- Samsung Galaxy Landscape (800×360, 3x)
27+
28+
### Tablets (Portrait)
29+
- iPad Mini (768×1024, 2x)
30+
- iPad Air (820×1180, 2x)
31+
- iPad Pro 11" (834×1194, 2x)
32+
- iPad Pro 12.9" (1024×1366, 2x)
33+
- Samsung Galaxy Tab (800×1280, 2x)
34+
35+
### Tablets (Landscape)
36+
- iPad Pro 11" Landscape (1194×834, 2x)
37+
- iPad Pro 12.9" Landscape (1366×1024, 2x)
38+
39+
### Desktop
40+
- HD (1366×768, 1x)
41+
- Full HD (1920×1080, 1x)
42+
- QHD (2560×1440, 1x)
43+
- 4K (3840×2160, 1x)
44+
45+
### Ultrawide
46+
- QHD Ultrawide (3440×1440, 1x)
47+
- 4K Ultrawide (5120×2160, 1x)
48+
49+
## Screens Captured
50+
51+
- **Landing Page** (`/`)
52+
- **Login** (`/login`)
53+
- **Workspace** (`/workspace`)
54+
- **Graph View** (`/graph`) - Core visualization
55+
- **Projects** (`/projects`)
56+
- **Settings** (`/settings`)
57+
- **Profile** (`/profile`)
58+
- **Admin Panel** (`/admin`)
59+
- **Admin Users** (`/admin/users`)
60+
- **Admin System** (`/admin/system`)
61+
62+
Plus interactive states:
63+
- Button hover states (up to 5 buttons)
64+
- Modal/dialog states (up to 3 modals)
65+
66+
## Running the Suite
67+
68+
### Standalone Execution
69+
```bash
70+
npm run test:e2e:visual
71+
```
72+
73+
### As Part of Full E2E Test Suite
74+
```bash
75+
npm run test:e2e
76+
# Or in VM:
77+
./tools/test-vm-e2e.sh
78+
```
79+
80+
### Disable Visual Regression in E2E Suite
81+
```bash
82+
RUN_VISUAL_REGRESSION=false ./tools/test-vm-e2e.sh
83+
```
84+
85+
## Output Structure
86+
87+
```
88+
test-artifacts/visual-regression/{timestamp}/
89+
├── iPhone-SE/
90+
│ ├── landing-page.png
91+
│ ├── login.png
92+
│ ├── workspace.png
93+
│ └── ...
94+
├── iPad-Pro-11/
95+
│ ├── landing-page.png
96+
│ └── ...
97+
├── Desktop-Full-HD/
98+
│ ├── landing-page.png
99+
│ └── ...
100+
├── SUMMARY.md
101+
└── ...
102+
```
103+
104+
Each test run creates a timestamped directory with:
105+
- Device-specific subdirectories
106+
- PNG screenshots for each screen
107+
- `SUMMARY.md` with test metadata and configuration
108+
109+
## Integration with GraphDone-DevOps
110+
111+
The visual regression suite is designed to provide comprehensive data for the GraphDone-DevOps repository to consume and analyze. It does NOT include a complex results viewer - that responsibility belongs to GraphDone-DevOps.
112+
113+
### Expected DevOps Integration:
114+
115+
1. **Automated Comparison**: Use tools like Pixelmatch or Percy for visual diff analysis
116+
2. **Artifact Storage**: Upload screenshots to S3/artifact storage for historical tracking
117+
3. **CI/CD Alerts**: Trigger notifications when visual changes exceed thresholds
118+
4. **Baseline Management**: Store approved screenshots as baselines for comparison
119+
5. **Reporting Dashboard**: Build viewing and organization tools in GraphDone-DevOps
120+
121+
### Data Format
122+
123+
Screenshots are organized by:
124+
- **Timestamp**: ISO format (YYYY-MM-DDTHH-mm-ss)
125+
- **Device**: Descriptive name (e.g., "iPhone-14-Pro-Max", "Desktop-Full-HD")
126+
- **Screen**: Sanitized route name (e.g., "landing-page", "admin-users")
127+
128+
All filenames are consistent and parseable for automated processing.
129+
130+
## Configuration
131+
132+
### Adjusting Device List
133+
134+
Edit `tests/e2e/visual-regression-suite.spec.ts`:
135+
136+
```typescript
137+
const DEVICES = [
138+
{ name: 'Custom-Device', width: 1024, height: 768, deviceScaleFactor: 1 },
139+
// ... add more devices
140+
];
141+
```
142+
143+
### Adjusting Screens
144+
145+
Edit the `SCREENS` array:
146+
147+
```typescript
148+
const SCREENS = [
149+
{ route: '/custom-route', name: 'custom-screen' },
150+
// ... add more screens
151+
];
152+
```
153+
154+
### Adjusting Timeouts
155+
156+
- **Page Load**: Line 141 - `timeout: 30000` (30 seconds)
157+
- **Content Wait**: Line 148 - `waitForTimeout(2000)` (2 seconds)
158+
- **Screenshot Retry**: Line 81 - `maxRetries = 3`
159+
160+
## Performance Considerations
161+
162+
### Test Duration
163+
- ~5-10 seconds per device configuration
164+
- Total runtime: ~3-5 minutes for all 21 devices
165+
166+
### Disk Usage
167+
- ~50-200KB per screenshot (depends on content)
168+
- ~250-300 screenshots per run
169+
- Total: ~15-60MB per test run
170+
171+
### Resource Requirements
172+
- Memory: ~2GB RAM for Playwright + browsers
173+
- CPU: Moderate (screenshot capture is CPU-intensive)
174+
- Disk I/O: Moderate (writing many PNG files)
175+
176+
## Best Practices
177+
178+
1. **Run on Stable State**: Execute after UI changes are complete
179+
2. **Consistent Environment**: Use same browser versions for comparisons
180+
3. **Network Independence**: Tests should not depend on external services
181+
4. **Baseline Updates**: Update baselines when intentional UI changes occur
182+
5. **Artifact Cleanup**: Regularly archive or delete old screenshot sets
183+
184+
## Troubleshooting
185+
186+
### Screenshots Failing
187+
- Check if application is running (`npm run dev`)
188+
- Verify routes exist in the application
189+
- Increase timeouts if content loads slowly
190+
191+
### Missing Browsers
192+
- Run `npx playwright install --with-deps`
193+
- Verify in VM: `ls -la ~/.cache/ms-playwright/`
194+
195+
### Incomplete Screenshot Sets
196+
- Check disk space
197+
- Review Playwright logs for specific errors
198+
- Verify network connectivity to localhost:3127
199+
200+
## Future Enhancements
201+
202+
- [ ] Add visual diff comparison tool integration
203+
- [ ] Implement baseline screenshot management
204+
- [ ] Add screenshot annotations (highlights, labels)
205+
- [ ] Support for authenticated routes
206+
- [ ] Dark mode screenshot variants
207+
- [ ] Accessibility contrast analysis
208+
- [ ] Mobile gesture simulation capture
209+
- [ ] Video recording for interactions
210+
211+
## Related Documentation
212+
213+
- E2E Test Suite: `tests/e2e/`
214+
- Test VM Setup: `tools/test-vm-e2e.sh`
215+
- Playwright Config: `playwright.config.ts`
216+
- DevOps Integration: (See GraphDone-DevOps repository)

‎tools/analyze-test-timing.sh‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
#!/bin/bash
2+
3+
# GraphDone Test Timing Analyzer
4+
# Analyzes test report logs and generates detailed timing breakdowns
5+
6+
set -e
7+
8+
REPORT_FILE="$1"
9+
10+
if [ -z "$REPORT_FILE" ] || [ ! -f "$REPORT_FILE" ]; then
11+
echo "Usage: $0 <test-report-file>"
12+
exit 1
13+
fi
14+
15+
echo "# Test Timing Analysis"
16+
echo ""
17+
echo "Analyzing: $REPORT_FILE"
18+
echo ""
19+
20+
# Extract timestamps from log files
21+
REPORT_DIR=$(dirname "$REPORT_FILE")
22+
23+
declare -A STEP_TIMES
24+
25+
# Parse build log for duration
26+
if [ -f "$REPORT_DIR/build.log" ]; then
27+
BUILD_TIME=$(grep -oP 'Done in \K[\d.]+s' "$REPORT_DIR/build.log" | tail -1 || echo "N/A")
28+
echo "Build Duration: $BUILD_TIME"
29+
fi
30+
31+
# Parse unit test log for duration
32+
if [ -f "$REPORT_DIR/unit-tests.log" ]; then
33+
TEST_TIME=$(grep -oP 'Test Files.*\(\K[\d.]+s' "$REPORT_DIR/unit-tests.log" | tail -1 || echo "N/A")
34+
echo "Unit Tests Duration: $TEST_TIME"
35+
fi
36+
37+
# Parse E2E test log for duration
38+
if [ -f "$REPORT_DIR/e2e-tests.log" ]; then
39+
E2E_TIME=$(grep -oP '\d+ passed.*\(\K[\d.]+s' "$REPORT_DIR/e2e-tests.log" | tail -1 || echo "N/A")
40+
echo "E2E Tests Duration: $E2E_TIME"
41+
fi
42+
43+
# Parse visual regression log for duration
44+
if [ -f "$REPORT_DIR/visual-regression.log" ]; then
45+
VR_TIME=$(grep -oP '\d+ passed.*\(\K[\d.]+s' "$REPORT_DIR/visual-regression.log" | tail -1 || echo "N/A")
46+
echo "Visual Regression Duration: $VR_TIME"
47+
fi
48+
49+
echo ""
50+
echo "## Detailed Breakdown"
51+
echo ""
52+
53+
# Analyze file modification times to estimate step durations
54+
cd "$REPORT_DIR"
55+
56+
if [ -f "vm-launch.log" ]; then
57+
VM_LAUNCH_START=$(stat -c %Y vm-launch.log 2>/dev/null || echo "0")
58+
fi
59+
60+
if [ -f "cloud-init.log" ]; then
61+
CLOUD_INIT_START=$(stat -c %Y cloud-init.log 2>/dev/null || echo "0")
62+
fi
63+
64+
if [ -f "lint.log" ]; then
65+
LINT_START=$(stat -c %Y lint.log 2>/dev/null || echo "0")
66+
fi
67+
68+
if [ -f "typecheck.log" ]; then
69+
TYPECHECK_START=$(stat -c %Y typecheck.log 2>/dev/null || echo "0")
70+
fi
71+
72+
if [ -f "build.log" ]; then
73+
BUILD_START=$(stat -c %Y build.log 2>/dev/null || echo "0")
74+
fi
75+
76+
if [ -f "unit-tests.log" ]; then
77+
UNIT_START=$(stat -c %Y unit-tests.log 2>/dev/null || echo "0")
78+
fi
79+
80+
if [ -f "e2e-tests.log" ]; then
81+
E2E_START=$(stat -c %Y e2e-tests.log 2>/dev/null || echo "0")
82+
fi
83+
84+
# Calculate durations from file timestamps
85+
if [ "$VM_LAUNCH_START" != "0" ] && [ "$CLOUD_INIT_START" != "0" ]; then
86+
LAUNCH_DURATION=$((CLOUD_INIT_START - VM_LAUNCH_START))
87+
echo "VM Launch: ${LAUNCH_DURATION}s"
88+
fi
89+
90+
if [ "$LINT_START" != "0" ] && [ "$TYPECHECK_START" != "0" ]; then
91+
LINT_DURATION=$((TYPECHECK_START - LINT_START))
92+
echo "Linting: ${LINT_DURATION}s"
93+
fi
94+
95+
if [ "$TYPECHECK_START" != "0" ] && [ "$BUILD_START" != "0" ]; then
96+
TYPECHECK_DURATION=$((BUILD_START - TYPECHECK_START))
97+
echo "Type Checking: ${TYPECHECK_DURATION}s"
98+
fi
99+
100+
if [ "$BUILD_START" != "0" ] && [ "$UNIT_START" != "0" ]; then
101+
BUILD_DURATION=$((UNIT_START - BUILD_START))
102+
echo "Build: ${BUILD_DURATION}s"
103+
fi
104+
105+
if [ "$UNIT_START" != "0" ] && [ "$E2E_START" != "0" ]; then
106+
UNIT_DURATION=$((E2E_START - UNIT_START))
107+
echo "Unit Tests: ${UNIT_DURATION}s"
108+
fi
109+
110+
echo ""
111+
echo "## Recommendations"
112+
echo ""
113+
114+
# Add intelligent recommendations based on timing
115+
if [ "$BUILD_DURATION" -gt 180 ] 2>/dev/null; then
116+
echo "- Build took >${BUILD_DURATION}s: Consider build caching or Turbo optimization"
117+
fi
118+
119+
if [ "$UNIT_DURATION" -gt 300 ] 2>/dev/null; then
120+
echo "- Unit tests took >${UNIT_DURATION}s: Consider test parallelization or filtering"
121+
fi
122+
123+
if [ -n "$E2E_TIME" ] && [ "$E2E_TIME" != "N/A" ]; then
124+
E2E_SECONDS=$(echo "$E2E_TIME" | sed 's/s//')
125+
if [ "$(echo "$E2E_SECONDS > 900" | bc)" -eq 1 ] 2>/dev/null; then
126+
echo "- E2E tests took >${E2E_SECONDS}s (15+ min): Consider parallelization or reducing scope"
127+
fi
128+
fi

0 commit comments

Comments
 (0)