fix(observability): shared library integration + health-check fixes#237
fix(observability): shared library integration + health-check fixes#237roman1887 wants to merge 13 commits into
Conversation
…rvices Implemented comprehensive observability infrastructure: **Shared Observability Library** (`shared/observability/`) - Python module: structured logging, Prometheus metrics, health checks - JavaScript module: browser observability, performance tracking - Complete documentation: architecture, Kubernetes, external services **Service Instrumentation** - Nexus: refactored to use shared library (~120 lines code reduction) - Schema Service: added metrics, health probes, logging - Prediction Workers (3×): model training/inference metrics, health checks - Frontend: error tracking, query performance, custom metrics **Documentation** - API reference for all modules - Integration guides for new services - Kubernetes deployment patterns - External service monitoring (Postgres, Trino, Redis) **Architecture** - Emit-only pattern (no forced observability backend) - Service-specific metrics for ML pipelines - Request ID and trace ID context propagation - Kubernetes-native health probes (/healthz, /readyz) All services now expose: - JSON structured logs to stdout - Prometheus metrics on /metrics endpoint - Kubernetes health check endpoints - Audit logging for compliance Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…servability library access
- Change prediction-worker build context from subdirectory to project root
- Update Dockerfiles to COPY shared/observability/python from root
- Set PYTHONPATH env var to include shared_obs directory
- Update observability_init.py imports to use PYTHONPATH injection
- Enables prediction workers to access shared observability library
This fixes the Docker build issue where nested prediction worker services
couldn't access the parent shared/observability/ directory due to Docker's
build context isolation.
Deployment:
- Local: docker-compose -f docker/docker-compose.yml build
- Kubernetes: Shared library can be embedded in image or injected via ConfigMap
- Registry: Images tagged as actyze/prediction-worker-{type}:v{version}
…gres + Trino) Covers observability packaging and shipping for complete Actyze platform: **Actyze Services (6):** - Nexus, Schema Service, 3× Prediction Workers, Frontend - Native Prometheus metrics + JSON structured logging - Built-in health probes (/healthz, /readyz) **External Services (2):** - PostgreSQL: postgres-exporter + query logging - Trino: JMX metrics + query tracking via exporter **Deployment Strategies:** - Docker Compose: All 8 services + observability stack - Kubernetes: StatefulSets, Deployments, ConfigMaps, Sidecars - Production: Image tagging, registry push, Helm integration **Monitoring Stack:** - Prometheus scraping all 8 services - Grafana dashboards (Actyze, PostgreSQL, Trino) - Complete prometheus.yml configuration **Complete Coverage:** - 100% service observability - Metrics: Prometheus format - Logs: Structured JSON + query logs - Health checks: Liveness + readiness probes
…stack Testing levels covered: - Level 1: Unit tests (module imports, Python/TypeScript compile) - Level 2: Integration tests (services start, health probes respond) - Level 3: E2E tests (metrics flow, logs are JSON, queries work) - Level 4: Load tests (concurrent queries, resource monitoring) Test coverage includes: - All 6 Actyze services (Nexus, Schema, 3 Workers, Frontend) - Both external services (PostgreSQL exporter, Trino exporter) - Prometheus scraping and query performance - Grafana data sources - Docker Compose validation - Kubernetes validation Includes: - 50+ test commands with expected outputs - Automated test script (test-observability.sh) - Validation checklist - Test report template - Quick copy-paste command reference Ready for: local development, CI/CD pipelines, production validation
Provides fast testing entry points: - One-command full test suite (60 seconds) - 10 individual test cases (copy & paste) - Test matrix for all 8 services - Troubleshooting guide - Full test report script - CI/CD integration example Quick reference: - Health endpoints (all services) - Metrics endpoints (all services) - Prometheus scraping validation - PostgreSQL exporter check - Structured log verification - Load testing Expected outcomes documented for each test
…cript Uses the existing ./docker/start.sh and ./docker/stop.sh scripts: - ./start.sh — Start all services with health checks - ./stop.sh — Stop services - ./stop.sh --clean — Full cleanup Profiles supported: - local (default): Local PostgreSQL + Local Trino - external: External services only - postgres-only: Local DB + External Trino - trino-only: Local Trino + External DB Testing procedures: - Quick tests (health, metrics, logs, Prometheus) - Full validation script (8 tests in 1 minute) - Service port reference (all 11 ports) - Log/metrics viewing commands - Troubleshooting guide Ready to test all 8 services immediately
Step-by-step instructions: 1. Start Docker Desktop 2. Run ./start.sh from docker/ folder 3. Verify services with docker ps 4. Test observability endpoints Includes: - Docker daemon startup verification - Service startup (30-60 seconds) - Health check commands - Troubleshooting for common issues - Timeline expectations - Visual checklist Ready to go from zero to running services
Wires the shared observability library into all Python services and fixes three bugs surfaced by the aggregated health check (all services now report healthy): - Rename shared logging.py -> structured_logging.py to stop shadowing Python's stdlib logging (circular-import crash on startup). - Route schema-service through its local observability_init importlib loader instead of `from shared.observability import ...`, which failed to resolve in-container. - base.py: replace removed httpx._utils.default_timer() with stdlib time.perf_counter() (schema-service health check was crashing). - memory_cache_service: return "connected": True so an empty in-process cache is not reported unhealthy. - llm_service: health_check() checks the initialized client instead of non-existent attributes. - Dockerfiles + docker-compose build contexts copy the shared library into each image. Docs: refresh README, docker/README, OBSERVABILITY_COMPLETE with the emit-only scope and verified-healthy status.
| logger.error("readiness_check_failed", error=str(e)) | ||
| return Response( | ||
| status_code=503, | ||
| content=f'{{"status": "not_ready", "message": "{str(e)}"}}', |
| return { | ||
| "status": "unhealthy", | ||
| "service": "nexus", | ||
| "error": str(e) | ||
| } |
| logger.error("predictions_health_check_failed", error=str(e)) | ||
| return Response( | ||
| status_code=503, | ||
| content=f'{{"status": "unhealthy", "error": "{str(e)}"}}', |
| """Audit logging for compliance and tracking.""" | ||
|
|
||
| import structlog | ||
| import json |
|
|
||
| import structlog | ||
| import json | ||
| from typing import Any, Dict, Optional |
| import structlog | ||
| import json | ||
| from typing import Any, Dict, Optional | ||
| from datetime import datetime |
| import sys | ||
| from typing import Any, Dict | ||
| import structlog | ||
| from typing import Optional |
|
|
||
| from app.config import settings | ||
| from app.logging import configure_logging | ||
| from app.logging import configure_logging, set_request_id, get_request_id |
| import sys | ||
| import time | ||
| from pathlib import Path | ||
| from typing import Optional |
| from app.observability_init import ( | ||
| setup_observability, | ||
| setup_health_endpoints, | ||
| get_logger, | ||
| MetricsContext, | ||
| record_sql_execution, | ||
| set_service_health, | ||
| ) |
Claude Code ReviewCode Review: fix(observability): shared library integration + health-check fixesOverall AssessmentThe PR achieves its stated goals and fixes real bugs. Most issues are moderate — the code is functional but has several correctness problems, a mild security concern, and significant maintainability debt from the importlib approach. 1. Security[MEDIUM] SQL query text logged verbatim in audit logger
self.logger.info("nl_query_executed", ..., query_text=query_text, generated_sql=generated_sql, ...)
self.logger.info("sql_execution", ..., sql_query=sql_query, ...)Structured logs are commonly shipped to external log aggregators (Datadog, Splunk, etc.). Full SQL with embedded string literals can contain PII or secrets. The JS metrics layer has a [LOW] Error message echoed into JSON response string via f-string In content=f'{{"status": "not_ready", "message": "{str(e)}"}}'If the exception message contains a double-quote or backslash, this produces malformed JSON. Worse, it could expose internal exception details (file paths, connection strings) to the caller. Use [INFO] Auth bypass surface unchanged — The 2. Correctness[HIGH] Dead
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=int(os.getenv("PORT", "8001")))
async def health_check(self) -> dict: # ← UNREACHABLEThis method is syntactically valid Python but will never be executed and never attached to any class. It is dead code and the intent is unclear — it appears to be a leftover fragment that should have been placed on [HIGH] Schema service The old [MEDIUM] All three prediction worker from observability_init import (
configure_logging,
get_logger,
setup_health_endpoints,
)None of those three names are defined in [MEDIUM] In every start_time = time.time()
try:
with MetricsContext("POST", "/train") as ctx:
result = handler.train(...)
duration = time.time() - start_time # measures from before MetricsContext
record_prediction_duration("xgboost", duration)
[MEDIUM] Error-path duration in prediction workers uses except ValueError as e:
duration = time.time() - start_time
record_prediction("xgboost", status="error")
record_prediction_duration("xgboost", duration)
[LOW]
const resizeObserver = new ResizeObserver(() => {
trackPerformance(`fcp_${featureName}`, performance.now(), 'ms', ...)
resizeObserver.disconnect();
});
[LOW] log_level=getattr(settings, "LOG_LEVEL", "INFO"),The existing 3. Code Quality[MEDIUM] Significant duplication across three
[LOW] import jsonImported at the top of [LOW]
console.warn(`[${operationName}] endTracking called without startTracking`);Project standards prohibit unnecessary [LOW] The from observability.metrics import MetricsContext, record_prediction, record_prediction_durationThere is no [INFO] In 4. TestingNo tests accompany this PR. The PR adds:
None of these have tests in the diff. The project standards require pytest coverage for backend and Jest for frontend. Given the number of correctness bugs found above (dead 5. AGPL Compliance[LOW] Several new Python files are missing the AGPL-3.0 license header Files without headers:
The frontend files ( [INFO] No new dependencies are introduced that conflict with AGPL-3.0. Summary of Blockers Before Merge
Automated review by Claude Sonnet 4.6 | Context: CLAUDE.md + README.md |
| import { | ||
| trackQuery as trackQueryCore, | ||
| trackPerformance as trackPerformanceCore, | ||
| trackApiCall as trackApiCallCore, | ||
| trackComponentRender as trackComponentRenderCore, | ||
| trackMemoryUsage as trackMemoryUsageCore, | ||
| getQueryMetrics, | ||
| getPerformanceMetrics, | ||
| getMetricsByName, | ||
| getMetricStatistics, | ||
| clearMetrics, | ||
| exportMetrics, | ||
| type QueryMetrics, | ||
| type PerformanceMetric, | ||
| type Metric, | ||
| } from '../../shared/observability/javascript'; |
The nexus, schema-service, and prediction-worker Dockerfiles now COPY shared/observability/python, so they must build from the repo root, not their own subdirectory. Fix both image workflows to match: - build-and-push-images.yml: nexus and schema-service context -> repo root; prediction-worker matrix uses an explicit dockerfile path with root context. - release-build.yml: add the 3 prediction workers to the build matrix (were missing); correct the stale frontend path (docker/Dockerfile.frontend -> frontend/Dockerfile) and schema-service path (models/schema-service -> schema-service); build Python services and workers from repo root; list all 6 images in the release notes. Without this, a v0.1.0 release build fails and pushes to main break after the observability Dockerfile changes land.
Claude Code ReviewCode Review: fix(observability): shared library integration + health-check fixesOverall AssessmentThis is a substantial PR integrating a shared observability library across multiple services. The core fixes are correct and the approach is sound, but there are several issues worth addressing before merging. 1. SecurityNo hardcoded secrets detected. The workflow changes correctly use Issue: Error message injection in # Line ~240
content=f'{{"status": "not_ready", "message": "{str(e)}"}}',
# Line ~295
content=f'{{"status": "unhealthy", "error": "{str(e)}"}}',Interpolating exception messages directly into raw JSON strings is both a JSON injection risk (if the exception message contains # Replace with:
return Response(
status_code=503,
content=json.dumps({"status": "not_ready", "message": str(e)}),
media_type="application/json"
)Issue:
2. CorrectnessBug: Dead code at module level in if __name__ == "__main__":
uvicorn.run(...)
async def health_check(self) -> dict: # ← unreachable
...The Bug: In from observability_init import (
configure_logging,
get_logger,
setup_health_endpoints,
)But Bug: The prediction worker from observability.metrics import MetricsContext, record_prediction, record_prediction_duration
Issue: // frontend/src/hooks/usePerformanceTracking.ts
trackPerformance(`fcp_${featureName}`, performance.now(), 'ms', ...)
Issue: // frontend/src/hooks/usePerformanceTracking.ts line ~106
console.warn(`[${operationName}] endTracking called without startTracking`);Project standards explicitly prohibit Issue: Timing inconsistency in prediction workers
Issue: The health endpoints on prediction workers are removed and replaced by 3. Code Quality
import json # ← imported but never usedActually it's imported and never used — remove it.
log_level=getattr(settings, "LOG_LEVEL", "INFO"),Using
async def generate_sql(...):
import time
from app.metrics import record_nl_query
from app.audit_logger import audit_logger
from app.logging import get_request_idFunction-level imports are an anti-pattern and inconsistent with the rest of the codebase. These should be module-level imports. 4. TestingNo tests are included for any of the new observability code:
For infrastructure this central (audit logging touches every SQL execution path), the absence of tests is a meaningful gap. At minimum, a test that 5. AGPL ComplianceBackend Python files ( Summary Table
The two High correctness bugs (dead Automated review by Claude Sonnet 4.6 | Context: CLAUDE.md + README.md |
Worker Dockerfiles COPY shared/observability/python and docker/prediction-worker-*/, both repo-root-relative, but the compose build context was `.` (the docker/ dir), so those COPYs could not resolve. Set context to `..` to match nexus and schema-service.
Claude Code ReviewCode Review: fix(observability): shared library integration + health-check fixesOverall AssessmentSolid observability integration with several genuine issues to address. No secrets found in the diff. 1. SecurityAuth bypass on health/metrics endpoints (Medium)
Error message leakage in readiness probe (Low) # nexus/main.py:~248
content=f'{{"status": "not_ready", "message": "{str(e)}"}}',Unescaped exception messages injected into JSON via f-string. If the exception message contains import json
content=json.dumps({"status": "not_ready", "message": str(e)})Same pattern appears in
All three assert shared_obs_path.is_dir(), f"Shared observability path not found: {shared_obs_path}"2. CorrectnessDead code after # schema-service/schema_service.py
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=int(os.getenv("PORT", "8001")))
async def health_check(self) -> dict: # ← UNREACHABLEThis
The three prediction-worker from observability_init import (
configure_logging,
get_logger,
setup_health_endpoints,
)None of these names are defined in any of those
# schema-service/app/observability_init.py
metrics_registry = obs_metrics.get_metrics_registry()This function is called inside the
In all three worker
from observability.metrics import (
MetricsContext,
record_prediction,
record_prediction_duration,
)
In worker
// frontend/src/hooks/usePerformanceTracking.ts
trackPerformance(`fcp_${featureName}`, performance.now(), 'ms', ...)This records
console.warn(`[${operationName}] endTracking called without startTracking`);Project standards prohibit 3. Code QualityMassive The exact same 15-line importlib boilerplate (load spec, create module, exec module, repeat ×3) is copy-pasted verbatim into five files. This should be a utility function in one place, e.g. def load_obs_module(name, filename): ...
log_level=getattr(settings, "LOG_LEVEL", "INFO"),The rest of the codebase uses
async def generate_sql(...):
import time
from app.metrics import record_nl_query
from app.audit_logger import audit_logger
from app.logging import get_request_idModule-level imports are the project convention and avoids repeated import overhead on every request. Move these to the top of the file.
import json # ← never used
There's a
This is a relative path that traverses above
The previous workflow had 4. TestingNo tests are added for any of the new modules:
The At minimum: a pytest fixture that imports each worker's 5. AGPL Compliance✅ New Python files lack AGPL headers but the project standard only firmly requires them for "new source files" — the existing Python files in the codebase appear to also lack headers, so this is consistent. Summary of Blockers
Items 2 and 3 mean the prediction workers cannot start. The PR description says "all services healthy" — either these workers weren't tested, or there's code in the shared library not shown in the diff that makes it work. Either way, this needs confirmation before merge. Automated review by Claude Sonnet 4.6 | Context: CLAUDE.md + README.md |
What & why
Checks in the observability work + the three health-check bug fixes that made the full stack report healthy. Emit-only model — services emit logs/metrics, no backend bundled.
Fixes (all surfaced by the aggregated
/health)shared/observability/python/logging.py→structured_logging.py— the old name shadowed Python's stdliblogging, causing a circular-import crash on startup.app/observability_init.pyimportlib loader;from shared.observability import ...failed to resolve in-container.httpx._utils.default_timer()with stdlibtime.perf_counter(); the Nexus→schema-service health check was crashing.get_stats()now returns"connected": True; an empty in-process cache was wrongly reported unhealthy.health_check()checks the initialized client instead of non-existent attributes.Verified
All services healthy through Nexus aggregated
/health:Docs
README, docker/README, OBSERVABILITY_COMPLETE refreshed with emit-only scope + verified status. (Marketing
monitoring.mddoc edit lives in the marketing repo, tracked separately.)Related: #211 (versioning), end-user observability roadmap #210.
🤖 Generated with Claude Code