Skip to content

perf(manager-server): accelerate usage analytics with hourly rollups#327

Merged
seakee merged 2 commits into
mainfrom
perf/usage-analytics-hourly-rollup
Jul 13, 2026
Merged

perf(manager-server): accelerate usage analytics with hourly rollups#327
seakee merged 2 commits into
mainfrom
perf/usage-analytics-hourly-rollup

Conversation

@seakee

@seakee seakee commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Summary

Reuse complete hourly rollups for eligible unfiltered Usage Analytics queries while preserving raw-event fallbacks for filters, percentiles, unsupported timezones, and incomplete checkpoints.
This reduces the 100k-event rollup-owned core from 777 ms / 23.70 MB to 39 ms / 9.51 MB, and publishes a bilingual report for the July 10 performance work.

Scope

  • Frontend panel
  • Manager Server
  • CPA panel mode
  • Full Docker mode
  • Native packages / release
  • Docs / Wiki
  • CI / build / tooling

Changes

  • Add a shared hourly-rollup reader and migrate Dashboard plus strictly unfiltered Usage Analytics core metrics to it.
  • Preserve raw-event parity for edge buckets, filtered or high-dimensional queries, P95/TTFT metrics, DST behavior, and unsupported timezone buckets.
  • Add parity, fallback, stability, race, and benchmark coverage plus bilingual performance documentation.

User Impact

Long-window Usage Analytics overview queries use materially less CPU time and memory when the hourly rollup is current. Filtered analytics behavior and API response semantics remain unchanged.

Compatibility / Runtime Notes

  • CPA panel mode: No behavior change.
  • Manager Server mode: Hourly rollup remains enabled by default and now serves eligible Dashboard and Usage Analytics reads.
  • Full Docker / native packages: No deployment change. Set USAGE_DASHBOARD_HOURLY_ROLLUP_ENABLED=false and restart Manager Server to force raw-event reads.

Data / Security Notes

No schema change and no new sensitive-data exposure. Existing rollup tables and sanitized API behavior are unchanged.

Risk / Rollback

Risk level: Medium

Rollback notes:

  • Set USAGE_DASHBOARD_HOURLY_ROLLUP_ENABLED=false and restart Manager Server.
  • The service will fall back to raw events without deleting existing rollup data.

Verification

  • Type check
  • Lint
  • Tests
  • Build
  • Manual UI check
  • Docs/link check
  • Not applicable, docs-only

Commands / evidence:

npm run type-check
npm run lint
npm run test                       # 86 files, 728 tests
npm --workspace apps/web run build:bundle
npm run check:demo-isolation
npm run docs:build
npm run manager-server:test
cd apps/manager-server && go test -race ./...
cd apps/manager-server && go vet ./...
git diff --check origin/main...HEAD

Benchmark evidence:

  • Overview: 3.21 s / 34.43 MB -> 2.48 s / 20.24 MB (~23% faster, ~41% lower B/op, ~64% fewer allocations).
  • Rollup-owned core: 777 ms / 23.70 MB -> 39 ms / 9.51 MB (~20x faster, ~60% lower B/op, ~92% fewer allocations).
  • Final heap profile: approximately 5 MB in use with no retained CPAMP reader/event slice; 200 iterations remained stable near 38-40 ms/op.

Screenshots / Recordings

N/A — backend and documentation changes only.

Docs

  • README updated
  • Wiki updated
  • Release notes needed
  • Not needed

Related

N/A

seakee added 2 commits July 10, 2026 23:31
Introduce a shared hourly-rollup reader and migrate Dashboard plus eligible unfiltered Usage Analytics metrics to it.

Preserve raw-event fallbacks for incomplete checkpoints, unsupported timezones, filters, percentiles, and high-dimensional metrics.

Adds parity, DST, fallback, stability, and benchmark coverage; runtime behavior remains reversible through the existing rollup switch.
Publish bilingual documentation covering the memory investigation, dashboard and analytics optimizations, and measured 100k-event benchmark results.

Document why each stage improves latency and allocations, how raw fallbacks preserve correctness, and how to disable the rollup path safely.

Add the report to both operations sidebars and link it from the Manager Server guide.
@seakee seakee merged commit 06a81c3 into main Jul 13, 2026
8 checks passed
@seakee seakee deleted the perf/usage-analytics-hourly-rollup branch July 13, 2026 03:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant