docs: add VERSIONING.md codifying SemVer + lockstep release strategy#236
docs: add VERSIONING.md codifying SemVer + lockstep release strategy#236roman1887 wants to merge 11 commits into
Conversation
…rvices Implemented comprehensive observability infrastructure: **Shared Observability Library** (`shared/observability/`) - Python module: structured logging, Prometheus metrics, health checks - JavaScript module: browser observability, performance tracking - Complete documentation: architecture, Kubernetes, external services **Service Instrumentation** - Nexus: refactored to use shared library (~120 lines code reduction) - Schema Service: added metrics, health probes, logging - Prediction Workers (3×): model training/inference metrics, health checks - Frontend: error tracking, query performance, custom metrics **Documentation** - API reference for all modules - Integration guides for new services - Kubernetes deployment patterns - External service monitoring (Postgres, Trino, Redis) **Architecture** - Emit-only pattern (no forced observability backend) - Service-specific metrics for ML pipelines - Request ID and trace ID context propagation - Kubernetes-native health probes (/healthz, /readyz) All services now expose: - JSON structured logs to stdout - Prometheus metrics on /metrics endpoint - Kubernetes health check endpoints - Audit logging for compliance Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…servability library access
- Change prediction-worker build context from subdirectory to project root
- Update Dockerfiles to COPY shared/observability/python from root
- Set PYTHONPATH env var to include shared_obs directory
- Update observability_init.py imports to use PYTHONPATH injection
- Enables prediction workers to access shared observability library
This fixes the Docker build issue where nested prediction worker services
couldn't access the parent shared/observability/ directory due to Docker's
build context isolation.
Deployment:
- Local: docker-compose -f docker/docker-compose.yml build
- Kubernetes: Shared library can be embedded in image or injected via ConfigMap
- Registry: Images tagged as actyze/prediction-worker-{type}:v{version}
…gres + Trino) Covers observability packaging and shipping for complete Actyze platform: **Actyze Services (6):** - Nexus, Schema Service, 3× Prediction Workers, Frontend - Native Prometheus metrics + JSON structured logging - Built-in health probes (/healthz, /readyz) **External Services (2):** - PostgreSQL: postgres-exporter + query logging - Trino: JMX metrics + query tracking via exporter **Deployment Strategies:** - Docker Compose: All 8 services + observability stack - Kubernetes: StatefulSets, Deployments, ConfigMaps, Sidecars - Production: Image tagging, registry push, Helm integration **Monitoring Stack:** - Prometheus scraping all 8 services - Grafana dashboards (Actyze, PostgreSQL, Trino) - Complete prometheus.yml configuration **Complete Coverage:** - 100% service observability - Metrics: Prometheus format - Logs: Structured JSON + query logs - Health checks: Liveness + readiness probes
…stack Testing levels covered: - Level 1: Unit tests (module imports, Python/TypeScript compile) - Level 2: Integration tests (services start, health probes respond) - Level 3: E2E tests (metrics flow, logs are JSON, queries work) - Level 4: Load tests (concurrent queries, resource monitoring) Test coverage includes: - All 6 Actyze services (Nexus, Schema, 3 Workers, Frontend) - Both external services (PostgreSQL exporter, Trino exporter) - Prometheus scraping and query performance - Grafana data sources - Docker Compose validation - Kubernetes validation Includes: - 50+ test commands with expected outputs - Automated test script (test-observability.sh) - Validation checklist - Test report template - Quick copy-paste command reference Ready for: local development, CI/CD pipelines, production validation
Provides fast testing entry points: - One-command full test suite (60 seconds) - 10 individual test cases (copy & paste) - Test matrix for all 8 services - Troubleshooting guide - Full test report script - CI/CD integration example Quick reference: - Health endpoints (all services) - Metrics endpoints (all services) - Prometheus scraping validation - PostgreSQL exporter check - Structured log verification - Load testing Expected outcomes documented for each test
…cript Uses the existing ./docker/start.sh and ./docker/stop.sh scripts: - ./start.sh — Start all services with health checks - ./stop.sh — Stop services - ./stop.sh --clean — Full cleanup Profiles supported: - local (default): Local PostgreSQL + Local Trino - external: External services only - postgres-only: Local DB + External Trino - trino-only: Local Trino + External DB Testing procedures: - Quick tests (health, metrics, logs, Prometheus) - Full validation script (8 tests in 1 minute) - Service port reference (all 11 ports) - Log/metrics viewing commands - Troubleshooting guide Ready to test all 8 services immediately
Step-by-step instructions: 1. Start Docker Desktop 2. Run ./start.sh from docker/ folder 3. Verify services with docker ps 4. Test observability endpoints Includes: - Docker daemon startup verification - Service startup (30-60 seconds) - Health check commands - Troubleshooting for common issues - Timeline expectations - Visual checklist Ready to go from zero to running services
Records the resolved versioning decisions (#211): strict SemVer, all Actyze images versioned in lockstep at one version == chart appVersion, 0.1.0 baseline, pin-never-float image policy, the release process, and the rule that the two Helm chart copies are kept in sync manually. Also documents the release-build.yml breakage to fix before v0.1.0. Refs #211
Claude Code ReviewPR Review: docs: add VERSIONING.md + observability instrumentation
1. Security 🔴SSRF / error message injection — The content=f'{{"status": "not_ready", "message": "{str(e)}"}}',If the exception message contains a double-quote, this produces malformed JSON. More importantly, it leaks internal exception text (stack details, hostnames, connection strings) to any unauthenticated caller—these are public probes. Use a fixed message or import json
content=json.dumps({"status": "not_ready", "message": "Service initialization failed"}),Same pattern in
This endpoint returns worker health status to anyone. It should be restricted, or at least confirm this is intentional. The old Credentials visible in health-check error logs — In logger.warning("postgres_health_check_failed", error="Connection refused") # not str(e)2. Correctness 🟡
All three worker copies contain: sys.path.insert(0, os.path.join(os.path.dirname(__file__), '../../shared'))
# ...imports happen here...
sys.path.insert(0, '/app/shared_obs')The second
import asyncio # inside setup_health_endpointsThis import is dead code in every copy.
The Same issue in Same fix: wrap in
The original {"last_updated": "...", "total_schemas": 42, "index_size": 1500}The replacement from
return Response(content=generate_latest(), ...)Without a registry argument this uses the default Prometheus registry, not
trackPerformance(`fcp_${featureName}`, performance.now(), 'ms', ...)
If 3. Code Quality 🟡Tripled All three copies are byte-for-byte identical (
import json
async def generate_sql(...):
import time
from app.metrics import record_nl_query
from app.audit_logger import audit_logger
from app.logging import get_request_idThese are module-level dependencies; import them at the top of the file. Deferred imports inside hot-path request handlers add overhead and obscure dependencies.
app_started = FalseA bare module-level boolean is not thread-safe and will not survive a reload or multi-worker deployment. Use an
const statusCode = (err as any)?.response?.status || 500;This bypasses TypeScript's type system. Define a typed error interface or use a type guard.
import { trackError } from '../utils/observability-init';
import { error as logError } from '../utils/observability-init';Both imports are from the same module. Combine into one import statement. 4. Testing 🟠No tests are added for any of the new code:
The project standards require tests for new features. The 5. AGPL Compliance ✅ / 🟡
Summary
Block on the SSRF/JSON-injection in Automated review by Claude Sonnet 4.6 | Context: CLAUDE.md + README.md |
| logger.error("readiness_check_failed", error=str(e)) | ||
| return Response( | ||
| status_code=503, | ||
| content=f'{{"status": "not_ready", "message": "{str(e)}"}}', |
| return { | ||
| "status": "unhealthy", | ||
| "service": "nexus", | ||
| "error": str(e) | ||
| } |
| logger.error("predictions_health_check_failed", error=str(e)) | ||
| return Response( | ||
| status_code=503, | ||
| content=f'{{"status": "unhealthy", "error": "{str(e)}"}}', |
| """Audit logging for compliance and tracking.""" | ||
|
|
||
| import structlog | ||
| import json |
|
|
||
| import structlog | ||
| import json | ||
| from typing import Any, Dict, Optional |
| import structlog | ||
| import json | ||
| from typing import Any, Dict, Optional | ||
| from datetime import datetime |
|
|
||
| from app.config import settings | ||
| from app.logging import configure_logging | ||
| from app.logging import configure_logging, set_request_id, get_request_id |
| import sys | ||
| import time | ||
| from pathlib import Path | ||
| from typing import Optional |
| health = HealthChecker() | ||
|
|
||
| # Register Trino health check | ||
| async def check_trino(trino_service=None): |
| from shared.observability import ( | ||
| get_logger, | ||
| MetricsContext, | ||
| record_sql_execution, | ||
| set_service_health, | ||
| ) |
| import { | ||
| trackQuery as trackQueryCore, | ||
| trackPerformance as trackPerformanceCore, | ||
| trackApiCall as trackApiCallCore, | ||
| trackComponentRender as trackComponentRenderCore, | ||
| trackMemoryUsage as trackMemoryUsageCore, | ||
| getQueryMetrics, | ||
| getPerformanceMetrics, | ||
| getMetricsByName, | ||
| getMetricStatistics, | ||
| clearMetrics, | ||
| exportMetrics, | ||
| type QueryMetrics, | ||
| type PerformanceMetric, | ||
| type Metric, | ||
| } from '../../shared/observability/javascript'; |
What & why
Codifies the versioning & release strategy resolved in #211 as the durable source of truth.
Decisions captured:
appVersion, bumped together0.1.0(pre-1.0)pullPolicy: IfNotPresent; neverlatest/main-llm-flexX.Y.Zimages → bumpappVersion+ pin → chart releasehelm-charts/(canonical, user-facing) anddashboard/.helm-charts/(DO demo copy) kept in sync manually; update both in the same change setAlso documents a blocker:
release-build.ymlomits the 3 prediction workers and points at two non-existent Dockerfile paths (docker/Dockerfile.frontend,models/schema-service/Dockerfile) — must fix before cuttingv0.1.0.Refs #211
🤖 Generated with Claude Code