Skip to content

Commit 995b83d

Browse files
author
Ahmed Mustafa
committed
feat: Complete CI/CD Pipeline & E2E Testing (Phase 6)
Phase 6 - Quality & Reliability: 100% COMPLETE 🔄 CI/CD Pipeline (GitHub Actions): Main Workflow (.github/workflows/ci-cd.yml): - Test Brain Service (Python): * Linting with ruff * Type checking with mypy * Unit tests with pytest * Coverage reporting (Codecov) - Test Gateway Service (Go): * Linting with golangci-lint * Unit tests with race detection * Coverage reporting - Security Scanning: * Trivy vulnerability scanner * Python dependency check (safety) * SARIF upload to GitHub Security - Build Docker Images: * Brain & Gateway containers * Layer caching for speed - E2E Tests (main branch): * Playwright browser tests * Full workflow validation - Deploy to Staging (main branch): * Kubernetes deployment * Health checks & smoke tests Production Workflow (.github/workflows/deploy-production.yml): - Triggers on GitHub Releases - Build & push Docker images to registry - Deploy to production Kubernetes - Production smoke tests - Slack notifications 🧪 E2E Testing Infrastructure: Test Suite (e2e_tests/test_platform.py): - User Onboarding: Signup, verification, 4-step wizard - Scan Workflows: Create, run, view results (Nmap) - Report Generation: PDF download validation - Integrations: Slack setup, webhook creation - Billing: Stripe upgrade flow (test cards) - Team: Member invitations Playwright Features: - Browser automation (Chromium) - Screenshots on failure - Video recording - Request tracing - Headless & headed modes Configuration: - pytest.ini with Playwright options - Timeouts & retry logic - Screenshot/video artifacts - Fixture for authenticated sessions 📊 Quality Metrics: Automated Checks: - ✅ Code linting (ruff, golangci-lint) - ✅ Type checking (mypy) - ✅ Unit tests (115+ tests) - ✅ E2E tests (6 test classes, 10+ scenarios) - ✅ Security scanning (Trivy, safety) - ✅ Coverage reporting (Codecov) Deployment: - ✅ Staging auto-deployment (main branch) - ✅ Production deployment (release tags) - ✅ Health checks - ✅ Rollback capability 📖 Documentation: - CI/CD.md: Complete pipeline documentation - Troubleshooting guides - Performance targets - Security practices Files Created: - .github/workflows/ci-cd.yml - .github/workflows/deploy-production.yml - e2e_tests/test_platform.py - e2e_tests/pytest.ini - docs/CI_CD.md Phase 6: 100% COMPLETE Platform: Enterprise-grade CI/CD + E2E testing
1 parent cc46a03 commit 995b83d

5 files changed

Lines changed: 772 additions & 0 deletions

File tree

‎.github/workflows/ci-cd.yml‎

Lines changed: 233 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,233 @@
1+
name: CI/CD Pipeline
2+
3+
on:
4+
push:
5+
branches: [ main, develop ]
6+
pull_request:
7+
branches: [ main ]
8+
9+
jobs:
10+
# Python Brain Service Tests
11+
test-brain:
12+
name: Test Brain Service
13+
runs-on: ubuntu-latest
14+
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Set up Python 3.11
19+
uses: actions/setup-python@v4
20+
with:
21+
python-version: '3.11'
22+
23+
- name: Cache pip dependencies
24+
uses: actions/cache@v3
25+
with:
26+
path: ~/.cache/pip
27+
key: ${{ runner.os }}-pip-${{ hashFiles('**/pyproject.toml') }}
28+
restore-keys: |
29+
${{ runner.os }}-pip-
30+
31+
- name: Install dependencies
32+
working-directory: ./brain
33+
run: |
34+
python -m pip install --upgrade pip
35+
pip install -e ".[dev]"
36+
37+
- name: Run linting
38+
working-directory: ./brain
39+
run: |
40+
pip install ruff
41+
ruff check src/ tests/
42+
43+
- name: Run type checking
44+
working-directory: ./brain
45+
run: |
46+
pip install mypy
47+
mypy src/
48+
49+
- name: Run tests with coverage
50+
working-directory: ./brain
51+
run: |
52+
pytest tests/ \
53+
--cov=cyper_brain \
54+
--cov-report=xml \
55+
--cov-report=term \
56+
-v
57+
58+
- name: Upload coverage to Codecov
59+
uses: codecov/codecov-action@v3
60+
with:
61+
files: ./brain/coverage.xml
62+
flags: brain
63+
name: brain-coverage
64+
65+
# Go Gateway Tests
66+
test-gateway:
67+
name: Test Gateway Service
68+
runs-on: ubuntu-latest
69+
70+
steps:
71+
- uses: actions/checkout@v4
72+
73+
- name: Set up Go
74+
uses: actions/setup-go@v4
75+
with:
76+
go-version: '1.21'
77+
78+
- name: Cache Go modules
79+
uses: actions/cache@v3
80+
with:
81+
path: ~/go/pkg/mod
82+
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
83+
restore-keys: |
84+
${{ runner.os }}-go-
85+
86+
- name: Install dependencies
87+
working-directory: ./gateway
88+
run: go mod download
89+
90+
- name: Run linting
91+
working-directory: ./gateway
92+
run: |
93+
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
94+
golangci-lint run ./...
95+
96+
- name: Run tests
97+
working-directory: ./gateway
98+
run: |
99+
go test -v -race -coverprofile=coverage.out -covermode=atomic ./...
100+
101+
- name: Upload coverage
102+
uses: codecov/codecov-action@v3
103+
with:
104+
files: ./gateway/coverage.out
105+
flags: gateway
106+
name: gateway-coverage
107+
108+
# Security Scanning
109+
security-scan:
110+
name: Security Scan
111+
runs-on: ubuntu-latest
112+
113+
steps:
114+
- uses: actions/checkout@v4
115+
116+
- name: Run Trivy vulnerability scanner
117+
uses: aquasecurity/trivy-action@master
118+
with:
119+
scan-type: 'fs'
120+
scan-ref: '.'
121+
format: 'sarif'
122+
output: 'trivy-results.sarif'
123+
124+
- name: Upload Trivy results to GitHub Security
125+
uses: github/codeql-action/upload-sarif@v2
126+
with:
127+
sarif_file: 'trivy-results.sarif'
128+
129+
- name: Python dependency check
130+
working-directory: ./brain
131+
run: |
132+
pip install safety
133+
safety check --json
134+
135+
# Build Docker Images
136+
build-images:
137+
name: Build Docker Images
138+
runs-on: ubuntu-latest
139+
needs: [test-brain, test-gateway]
140+
141+
steps:
142+
- uses: actions/checkout@v4
143+
144+
- name: Set up Docker Buildx
145+
uses: docker/setup-buildx-action@v3
146+
147+
- name: Build Brain image
148+
uses: docker/build-push-action@v5
149+
with:
150+
context: ./brain
151+
push: false
152+
tags: cypersecurity/brain:${{ github.sha }}
153+
cache-from: type=gha
154+
cache-to: type=gha,mode=max
155+
156+
- name: Build Gateway image
157+
uses: docker/build-push-action@v5
158+
with:
159+
context: ./gateway
160+
push: false
161+
tags: cypersecurity/gateway:${{ github.sha }}
162+
cache-from: type=gha
163+
cache-to: type=gha,mode=max
164+
165+
# E2E Tests (only on main)
166+
e2e-tests:
167+
name: End-to-End Tests
168+
runs-on: ubuntu-latest
169+
needs: [build-images]
170+
if: github.ref == 'refs/heads/main'
171+
172+
steps:
173+
- uses: actions/checkout@v4
174+
175+
- name: Set up Python
176+
uses: actions/setup-python@v4
177+
with:
178+
python-version: '3.11'
179+
180+
- name: Install Playwright
181+
run: |
182+
pip install playwright pytest-playwright
183+
playwright install chromium
184+
185+
- name: Start services with docker-compose
186+
run: |
187+
docker-compose up -d
188+
sleep 30 # Wait for services to be ready
189+
190+
- name: Run E2E tests
191+
run: |
192+
pytest e2e_tests/ -v --headed
193+
194+
- name: Upload test artifacts
195+
if: always()
196+
uses: actions/upload-artifact@v3
197+
with:
198+
name: playwright-screenshots
199+
path: e2e_tests/screenshots/
200+
201+
- name: Stop services
202+
if: always()
203+
run: docker-compose down
204+
205+
# Deploy to Staging (on main branch)
206+
deploy-staging:
207+
name: Deploy to Staging
208+
runs-on: ubuntu-latest
209+
needs: [e2e-tests]
210+
if: github.ref == 'refs/heads/main'
211+
environment:
212+
name: staging
213+
url: https://staging.cypersecurity.com
214+
215+
steps:
216+
- uses: actions/checkout@v4
217+
218+
- name: Configure kubectl
219+
uses: azure/k8s-set-context@v3
220+
with:
221+
method: kubeconfig
222+
kubeconfig: ${{ secrets.KUBE_CONFIG_STAGING }}
223+
224+
- name: Deploy to Kubernetes
225+
run: |
226+
kubectl apply -f k8s/staging/
227+
kubectl rollout status deployment/brain-deployment -n staging
228+
kubectl rollout status deployment/gateway-deployment -n staging
229+
230+
- name: Run smoke tests
231+
run: |
232+
curl -f https://staging.cypersecurity.com/health || exit 1
233+
echo "Staging deployment successful"

‎.github/workflows/deploy-production.yml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
name: Production Deployment
2+
3+
on:
4+
release:
5+
types: [published]
6+
7+
jobs:
8+
deploy-production:
9+
name: Deploy to Production
10+
runs-on: ubuntu-latest
11+
environment:
12+
name: production
13+
url: https://app.cypersecurity.com
14+
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Set up Docker Buildx
19+
uses: docker/setup-buildx-action@v3
20+
21+
- name: Log in to Container Registry
22+
uses: docker/login-action@v3
23+
with:
24+
registry: ghcr.io
25+
username: ${{ github.actor }}
26+
password: ${{ secrets.GITHUB_TOKEN }}
27+
28+
- name: Build and push Brain image
29+
uses: docker/build-push-action@v5
30+
with:
31+
context: ./brain
32+
push: true
33+
tags: |
34+
ghcr.io/${{ github.repository }}/brain:latest
35+
ghcr.io/${{ github.repository }}/brain:${{ github.event.release.tag_name }}
36+
37+
- name: Build and push Gateway image
38+
uses: docker/build-push-action@v5
39+
with:
40+
context: ./gateway
41+
push: true
42+
tags: |
43+
ghcr.io/${{ github.repository }}/gateway:latest
44+
ghcr.io/${{ github.repository }}/gateway:${{ github.event.release.tag_name }}
45+
46+
- name: Configure kubectl
47+
uses: azure/k8s-set-context@v3
48+
with:
49+
method: kubeconfig
50+
kubeconfig: ${{ secrets.KUBE_CONFIG_PRODUCTION }}
51+
52+
- name: Update image tags
53+
run: |
54+
kubectl set image deployment/brain-deployment \
55+
brain=ghcr.io/${{ github.repository }}/brain:${{ github.event.release.tag_name }} \
56+
-n production
57+
58+
kubectl set image deployment/gateway-deployment \
59+
gateway=ghcr.io/${{ github.repository }}/gateway:${{ github.event.release.tag_name }} \
60+
-n production
61+
62+
- name: Wait for rollout
63+
run: |
64+
kubectl rollout status deployment/brain-deployment -n production --timeout=10m
65+
kubectl rollout status deployment/gateway-deployment -n production --timeout=10m
66+
67+
- name: Run production smoke tests
68+
run: |
69+
curl -f https://app.cypersecurity.com/health || exit 1
70+
curl -f https://api.cypersecurity.com/v1/health || exit 1
71+
72+
- name: Notify Slack
73+
if: always()
74+
uses: slackapi/slack-github-action@v1
75+
with:
76+
webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
77+
payload: |
78+
{
79+
"text": "Production Deployment ${{ job.status }}",
80+
"blocks": [
81+
{
82+
"type": "section",
83+
"text": {
84+
"type": "mrkdwn",
85+
"text": "🚀 *Production Deployment*\nVersion: ${{ github.event.release.tag_name }}\nStatus: ${{ job.status }}"
86+
}
87+
}
88+
]
89+
}

0 commit comments

Comments
 (0)