Skip to content

docs: add VERSIONING.md codifying SemVer + lockstep release strategy#236

Open
roman1887 wants to merge 11 commits into
mainfrom
docs/versioning-strategy
Open

docs: add VERSIONING.md codifying SemVer + lockstep release strategy#236
roman1887 wants to merge 11 commits into
mainfrom
docs/versioning-strategy

Conversation

@roman1887

Copy link
Copy Markdown
Collaborator

What & why

Codifies the versioning & release strategy resolved in #211 as the durable source of truth.

Decisions captured:

  • Strict SemVer with Actyze-specific major/minor/patch definitions
  • Lockstep — all 6 Actyze images share one version == chart appVersion, bumped together
  • Baseline 0.1.0 (pre-1.0)
  • Pin, never float — user-facing chart + compose pin to immutable version tags, pullPolicy: IfNotPresent; never latest/main-llm-flex
  • Release process — Conventional Commits → release-drafter → publish tag → CI builds/pushes X.Y.Z images → bump appVersion + pin → chart release
  • Two-chart sync rulehelm-charts/ (canonical, user-facing) and dashboard/.helm-charts/ (DO demo copy) kept in sync manually; update both in the same change set

Also documents a blocker: release-build.yml omits the 3 prediction workers and points at two non-existent Dockerfile paths (docker/Dockerfile.frontend, models/schema-service/Dockerfile) — must fix before cutting v0.1.0.

Refs #211

🤖 Generated with Claude Code

roman1887 and others added 11 commits June 15, 2026 00:06
…rvices

Implemented comprehensive observability infrastructure:

**Shared Observability Library** (`shared/observability/`)
- Python module: structured logging, Prometheus metrics, health checks
- JavaScript module: browser observability, performance tracking
- Complete documentation: architecture, Kubernetes, external services

**Service Instrumentation**
- Nexus: refactored to use shared library (~120 lines code reduction)
- Schema Service: added metrics, health probes, logging
- Prediction Workers (3×): model training/inference metrics, health checks
- Frontend: error tracking, query performance, custom metrics

**Documentation**
- API reference for all modules
- Integration guides for new services
- Kubernetes deployment patterns
- External service monitoring (Postgres, Trino, Redis)

**Architecture**
- Emit-only pattern (no forced observability backend)
- Service-specific metrics for ML pipelines
- Request ID and trace ID context propagation
- Kubernetes-native health probes (/healthz, /readyz)

All services now expose:
- JSON structured logs to stdout
- Prometheus metrics on /metrics endpoint
- Kubernetes health check endpoints
- Audit logging for compliance

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…servability library access

- Change prediction-worker build context from subdirectory to project root
- Update Dockerfiles to COPY shared/observability/python from root
- Set PYTHONPATH env var to include shared_obs directory
- Update observability_init.py imports to use PYTHONPATH injection
- Enables prediction workers to access shared observability library

This fixes the Docker build issue where nested prediction worker services
couldn't access the parent shared/observability/ directory due to Docker's
build context isolation.

Deployment:
- Local: docker-compose -f docker/docker-compose.yml build
- Kubernetes: Shared library can be embedded in image or injected via ConfigMap
- Registry: Images tagged as actyze/prediction-worker-{type}:v{version}
…gres + Trino)

Covers observability packaging and shipping for complete Actyze platform:

**Actyze Services (6):**
- Nexus, Schema Service, 3× Prediction Workers, Frontend
- Native Prometheus metrics + JSON structured logging
- Built-in health probes (/healthz, /readyz)

**External Services (2):**
- PostgreSQL: postgres-exporter + query logging
- Trino: JMX metrics + query tracking via exporter

**Deployment Strategies:**
- Docker Compose: All 8 services + observability stack
- Kubernetes: StatefulSets, Deployments, ConfigMaps, Sidecars
- Production: Image tagging, registry push, Helm integration

**Monitoring Stack:**
- Prometheus scraping all 8 services
- Grafana dashboards (Actyze, PostgreSQL, Trino)
- Complete prometheus.yml configuration

**Complete Coverage:**
- 100% service observability
- Metrics: Prometheus format
- Logs: Structured JSON + query logs
- Health checks: Liveness + readiness probes
…stack

Testing levels covered:
- Level 1: Unit tests (module imports, Python/TypeScript compile)
- Level 2: Integration tests (services start, health probes respond)
- Level 3: E2E tests (metrics flow, logs are JSON, queries work)
- Level 4: Load tests (concurrent queries, resource monitoring)

Test coverage includes:
- All 6 Actyze services (Nexus, Schema, 3 Workers, Frontend)
- Both external services (PostgreSQL exporter, Trino exporter)
- Prometheus scraping and query performance
- Grafana data sources
- Docker Compose validation
- Kubernetes validation

Includes:
- 50+ test commands with expected outputs
- Automated test script (test-observability.sh)
- Validation checklist
- Test report template
- Quick copy-paste command reference

Ready for: local development, CI/CD pipelines, production validation
Provides fast testing entry points:
- One-command full test suite (60 seconds)
- 10 individual test cases (copy & paste)
- Test matrix for all 8 services
- Troubleshooting guide
- Full test report script
- CI/CD integration example

Quick reference:
- Health endpoints (all services)
- Metrics endpoints (all services)
- Prometheus scraping validation
- PostgreSQL exporter check
- Structured log verification
- Load testing

Expected outcomes documented for each test
…cript

Uses the existing ./docker/start.sh and ./docker/stop.sh scripts:
- ./start.sh — Start all services with health checks
- ./stop.sh — Stop services
- ./stop.sh --clean — Full cleanup

Profiles supported:
- local (default): Local PostgreSQL + Local Trino
- external: External services only
- postgres-only: Local DB + External Trino
- trino-only: Local Trino + External DB

Testing procedures:
- Quick tests (health, metrics, logs, Prometheus)
- Full validation script (8 tests in 1 minute)
- Service port reference (all 11 ports)
- Log/metrics viewing commands
- Troubleshooting guide

Ready to test all 8 services immediately
Step-by-step instructions:
1. Start Docker Desktop
2. Run ./start.sh from docker/ folder
3. Verify services with docker ps
4. Test observability endpoints

Includes:
- Docker daemon startup verification
- Service startup (30-60 seconds)
- Health check commands
- Troubleshooting for common issues
- Timeline expectations
- Visual checklist

Ready to go from zero to running services
Records the resolved versioning decisions (#211): strict SemVer, all
Actyze images versioned in lockstep at one version == chart appVersion,
0.1.0 baseline, pin-never-float image policy, the release process, and
the rule that the two Helm chart copies are kept in sync manually.
Also documents the release-build.yml breakage to fix before v0.1.0.

Refs #211
@github-actions

Copy link
Copy Markdown

Claude Code Review

PR Review: docs: add VERSIONING.md + observability instrumentation

Note: The PR title says "add VERSIONING.md" but the diff contains no VERSIONING.md. This is a substantial observability instrumentation PR bundled under a misleading title. The review covers what was actually changed.


1. Security 🔴

SSRF / error message injection — nexus/main.py lines ~253–261 and ~295–303

The f-string error messages in both /readyz and /api/health/predictions interpolate str(e) directly into a raw JSON string returned to the client:

content=f'{{"status": "not_ready", "message": "{str(e)}"}}',

If the exception message contains a double-quote, this produces malformed JSON. More importantly, it leaks internal exception text (stack details, hostnames, connection strings) to any unauthenticated caller—these are public probes. Use a fixed message or json.dumps to serialize safely:

import json
content=json.dumps({"status": "not_ready", "message": "Service initialization failed"}),

Same pattern in /api/health/predictions. Strip str(e) from both responses.

/metrics endpoint is unauthenticated — nexus/main.py

GET /metrics now exposes Prometheus metrics without any auth check. Metrics can reveal internal topology, query patterns, user counts, and LLM model names. Either place it behind require_viewer or restrict it to an internal-only port/path at the network layer. At minimum, document the exposure.

/api/health/predictions is unauthenticated — nexus/main.py

This endpoint returns worker health status to anyone. It should be restricted, or at least confirm this is intentional. The old /health was similarly open, but this one exposes more service topology.

Credentials visible in health-check error logs — observability_init.py (all three workers)

In check_postgres(), a caught exception is logged with error=str(e). psycopg2 exceptions routinely embed the DSN (host, port, database, user) in the message. This will write connection-string fragments to structured logs. Sanitize before logging:

logger.warning("postgres_health_check_failed", error="Connection refused")  # not str(e)

2. Correctness 🟡

observability_init.py — duplicate, conflicting sys.path manipulation

All three worker copies contain:

sys.path.insert(0, os.path.join(os.path.dirname(__file__), '../../shared'))
# ...imports happen here...
sys.path.insert(0, '/app/shared_obs')

The second sys.path.insert happens after the imports that depend on the first path. The /app/shared_obs path is a hardcoded container-internal path that silently does nothing in dev but shadows modules in production. This is fragile and the intent is unclear. Remove it or document it.

asyncio imported but never used — all three observability_init.py

import asyncio  # inside setup_health_endpoints

This import is dead code in every copy.

check_trino() in health checks uses synchronous trino.dbapi calls inside an async def

The cursor.execute("SELECT 1") call is blocking. On a slow or unreachable Trino this will block the event loop during a readiness probe, potentially cascading into timeouts. Wrap in asyncio.to_thread() or use a true async Trino client.

Same issue in check_postgres()psycopg2.connect() is blocking

Same fix: wrap in asyncio.to_thread().

schema-service/schema_service.py — old /health endpoint removed, but its data is gone

The original /health returned operationally useful data:

{"last_updated": "...", "total_schemas": 42, "index_size": 1500}

The replacement from setup_health_endpoints() returns only {"status": "healthy"}. This is a regression for operators monitoring schema-service state. The rich data should be preserved in /health or /ready.

nexus/main.pygenerate_latest() called without registry argument

return Response(content=generate_latest(), ...)

Without a registry argument this uses the default Prometheus registry, not metrics_registry imported from app.metrics. Inconsistent with how the worker observability_init.py files call generate_latest(metrics_registry). Pass the registry explicitly.

useFirstContentfulPaint hook measures wall time, not FCP

trackPerformance(`fcp_${featureName}`, performance.now(), 'ms', ...)

performance.now() here is milliseconds since navigation start, not duration since the element appeared. The metric name says FCP but the value is an absolute timestamp. Use performance.now() - startTimeRef.current or the PerformanceObserver API for real FCP.

MetricsContext used in async endpoints but appears to be a context manager wrapping synchronous handler.train() calls

If MetricsContext.__exit__ swallows exceptions before the except ValueError clause runs, error-path metrics will never be recorded. The pattern records metrics inside the with block on success, then records them again in the except blocks—this is a double-record on error if the context manager doesn't re-raise immediately. Verify MetricsContext re-raises all exceptions.


3. Code Quality 🟡

Tripled observability_init.py — identical files in three worker directories

All three copies are byte-for-byte identical (index 0000000..3315c55 on all three). This is the anti-pattern the shared observability/ library was created to avoid. Extract to shared/observability/python/worker_init.py (it already imports from there) and import it from one place. The current approach guarantees drift.

nexus/app/audit_logger.py — unused imports

import json

json is imported but never used. Remove it.

nexus/app/api.py — imports inside function bodies

async def generate_sql(...):
    import time
    from app.metrics import record_nl_query
    from app.audit_logger import audit_logger
    from app.logging import get_request_id

These are module-level dependencies; import them at the top of the file. Deferred imports inside hot-path request handlers add overhead and obscure dependencies.

nexus/main.py — global mutable state for readiness

app_started = False

A bare module-level boolean is not thread-safe and will not survive a reload or multi-worker deployment. Use an asyncio.Event or the existing orchestration_service health check as the authoritative source of readiness.

frontend/src/utils/observability-init.tsnavigator.platform is deprecated

navigator.platform has been deprecated in all major browsers. Remove it from the startup debug log.

useQueryObservability.tsany cast for status code

const statusCode = (err as any)?.response?.status || 500;

This bypasses TypeScript's type system. Define a typed error interface or use a type guard.

frontend/src/components/ErrorBoundary.tsx — duplicate import

import { trackError } from '../utils/observability-init';
import { error as logError } from '../utils/observability-init';

Both imports are from the same module. Combine into one import statement.


4. Testing 🟠

No tests are added for any of the new code:

  • No pytest tests for audit_logger.py, metrics.py, or the new middleware in nexus/main.py
  • No Jest tests for ErrorBoundary, useQueryObservability, or usePerformanceTracking
  • The health endpoint changes (/healthz, /readyz, /health deprecation) have no test coverage

The project standards require tests for new features. The /readyz JSON-injection bug above would have been caught by a test.


5. AGPL Compliance ✅ / 🟡

  • New Python files (audit_logger.py, metrics.py, logging.py, workers' observability_init.py) are missing the # SPDX-License-Identifier: AGPL-3.0-only header required by project standards. The Python worker observability_init.py files have it; the nexus files do not.
  • shared/observability/__init__.py has no license header.
  • schema-service/app/observability_init.py has no license header.
  • All new TypeScript/JS files correctly include the SPDX header. ✅
  • No incompatible dependencies introduced. ✅

Summary

Area Status
Security 🔴 SSRF/injection in error responses, unauthenticated metrics, credential leak in health logs
Correctness 🟡 Blocking I/O in async probes, FCP metric is wrong, registry mismatch in /metrics
Code Quality 🟡 Tripled identical files, imports inside hot path, JSON import unused
Testing 🟠 No tests added for any new code
AGPL headers 🟡 Missing on several new Python files

Block on the SSRF/JSON-injection in /readyz and /api/health/predictions before merge. The blocking I/O in health probes and the tripled observability_init.py should be addressed in this PR rather than deferred.


Automated review by Claude Sonnet 4.6 | Context: CLAUDE.md + README.md

Comment thread nexus/main.py
logger.error("readiness_check_failed", error=str(e))
return Response(
status_code=503,
content=f'{{"status": "not_ready", "message": "{str(e)}"}}',
Comment thread nexus/main.py
Comment on lines +248 to +252
return {
"status": "unhealthy",
"service": "nexus",
"error": str(e)
}
Comment thread nexus/main.py
logger.error("predictions_health_check_failed", error=str(e))
return Response(
status_code=503,
content=f'{{"status": "unhealthy", "error": "{str(e)}"}}',
Comment thread nexus/app/audit_logger.py
"""Audit logging for compliance and tracking."""

import structlog
import json
Comment thread nexus/app/audit_logger.py

import structlog
import json
from typing import Any, Dict, Optional
Comment thread nexus/app/audit_logger.py
import structlog
import json
from typing import Any, Dict, Optional
from datetime import datetime
Comment thread nexus/main.py

from app.config import settings
from app.logging import configure_logging
from app.logging import configure_logging, set_request_id, get_request_id
import sys
import time
from pathlib import Path
from typing import Optional
health = HealthChecker()

# Register Trino health check
async def check_trino(trino_service=None):
Comment on lines +21 to +26
from shared.observability import (
get_logger,
MetricsContext,
record_sql_execution,
set_service_health,
)
Comment on lines +20 to +35
import {
trackQuery as trackQueryCore,
trackPerformance as trackPerformanceCore,
trackApiCall as trackApiCallCore,
trackComponentRender as trackComponentRenderCore,
trackMemoryUsage as trackMemoryUsageCore,
getQueryMetrics,
getPerformanceMetrics,
getMetricsByName,
getMetricStatistics,
clearMetrics,
exportMetrics,
type QueryMetrics,
type PerformanceMetric,
type Metric,
} from '../../shared/observability/javascript';
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants