Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions README.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion benchmarks/AGENT_IMPACT.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WaveMind Agent Impact Leaderboard

Generated: `2026-07-11T03:37:56Z`.
Generated: `2026-07-11T05:56:44Z`.

Agent-impact rows come from checked-in benchmark artifacts. They show behavioral lift on the configured tasks; they do not claim general agent success outside the listed scenarios.

Expand Down
6 changes: 3 additions & 3 deletions benchmarks/BENCHMARK_LEADERBOARD.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# WaveMind Benchmark Leaderboard

Generated from `benchmarks/benchmark_matrix_results.json`.
Last refresh: `2026-07-11T03:37:56Z` from `be2cebfd776b`.
Last refresh: `2026-07-11T05:56:44Z` from `c4f786e131c8`.

This is a compact reader-facing view of checked-in benchmark results. It is not a universal vector-database leaderboard: each row uses the primary quality metric for that benchmark, and latency is shown separately so quality wins are not confused with speed wins.

Expand Down Expand Up @@ -34,7 +34,7 @@ This is a compact reader-facing view of checked-in benchmark results. It is not

| area | current source | claim status | next action |
|---|---|---|---|
| Artifact freshness | local matrix refresh at `2026-07-11T03:37:56Z` | source `be2cebfd776b`; audit gate enforced by `validate_benchmark_artifacts.py` | Keep weekly refresh green before public claims. |
| Artifact freshness | local matrix refresh at `2026-07-11T05:56:44Z` | source `c4f786e131c8`; audit gate enforced by `validate_benchmark_artifacts.py` | Keep weekly refresh green before public claims. |
| Serverless telemetry | loopback API pool; `loopback-api-capacity-estimate`; 4 measured replicas | observed SLO `True`; loopback evidence, not a managed-serverless claim | Run `.github/workflows/serverless-observed-telemetry.yml` against deployed API nodes. |
| External HTTP cluster load | kubernetes-kind-non-loopback-ci; `kubernetes-pod-dns-physical-node-drill`; 4 nodes | SLO `True`; non-loopback Kubernetes pod-DNS evidence | Run `.github/workflows/external-http-cluster-load.yml` with a remote node manifest. |
| External HTTP active-active loopback | local-loopback; `loopback-api-regions`; 3 regions | SLO `True`; external URL contract over local API regions | Run `.github/workflows/external-http-active-active.yml` with remote regions for production evidence. |
Expand All @@ -43,7 +43,7 @@ This is a compact reader-facing view of checked-in benchmark results. It is not
| 10M streaming load | local `WaveMind faiss-ivfpq-persisted streaming` profile | target recall `0.99`, p99 `60.1 ms`, SLO `scale_required` | Repeat at 50M and add service-backed Qdrant/pgvector 10M artifacts. |
| 50M streaming preflight | `WaveMind faiss-ivfpq-persisted streaming` plan-only contract | `action_required`; index `1.12 GB`; app storage `119.2 GB`; blockers `missing_env:WAVEMIND_FAISS_IVFPQ_PATH, insufficient_local_disk_for_index_and_transient_batches` | Run `.github/workflows/production-streaming-load.yml` with `faiss-ivfpq-persisted` and publish `benchmarks/production_streaming_load_ivfpq_50m_results.json`. |
| Qdrant streaming | real Qdrant service smoke plus measured 10M profile | smoke recall `1`, smoke p99 `17.9 ms`; 10M recall `0.975`, 10M p99 `43.3 ms`, SLO `scale_required` | Keep the measured 10M profile green and run the sharded Qdrant and pgvector 10M profiles next. |
| Qdrant sharded streaming | real two-service fanout smoke plus horizontal Qdrant preflight | smoke recall `1`, smoke p99 `16.0 ms`; 10M preflight `action_required`; 100M preflight `action_required`; planned shards `4`; blockers `missing_env:WAVEMIND_QDRANT_URLS, insufficient_local_disk_for_index_and_transient_batches` | Run `.github/workflows/production-streaming-load.yml` with `qdrant-sharded-service` and publish `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` or `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json`. |
| Qdrant sharded streaming | real fanout smoke plus measured four-service 10M profile | smoke recall `1`, smoke p99 `16.0 ms`; 10M recall `0.993`, 10M p99 `71.3 ms`, shards `4`; 100M preflight `action_required`; planned shards `4`; blockers `none (measured artifact passes)` | Keep the measured 10M sharded profile green and run the strict 100M sharded profile next. |
| Qdrant 1M streaming | real Qdrant service run before and after warmup/chunking tuning | cold p99 `3014.0 ms`; tuned recall `1`, tuned p99 `26.4 ms`, SLO `pass` | Use the tuned warmup/chunking profile for the 10M Qdrant service run. |
| pgvector streaming | real PostgreSQL/pgvector service smoke plus 10M preflight | smoke recall `1`, smoke p99 `7.624 ms`; 10M preflight `action_required` | Run `.github/workflows/production-streaming-load.yml` with `pgvector-service` against sized Postgres storage. |
| Production readiness gate | checked-in benchmark artifacts | `pass`; 39/39 pass | Keep the gate at readiness_score 1.0 while repeating larger service-backed runs and moving external competitor evidence into the separate adapter profile. |
Expand Down
6 changes: 3 additions & 3 deletions benchmarks/BENCHMARK_REPORT.md

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions benchmarks/COST_EFFICIENCY.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# WaveMind Cost Efficiency Leaderboard

Generated: `2026-07-11T03:37:56Z`.
Generated: `2026-07-11T05:56:44Z`.

Measured rows come from checked-in load artifacts. Planned rows are capacity and cost contracts only; they do not unlock production latency or recall claims until the matching benchmark result exists.

## Summary

- Measured rows: `21`.
- Measured rows: `22`.
- Measured SLO pass rows: `9`.
- Measured valid cost rows: `12`.
- Measured valid cost rows: `13`.
- Planned cost rows: `5`.
- Measured frontier: `sub_100k-wavemind-numpy-streaming-production_streaming_load_smoke_results, 100k-qdrant-service-production_load_qdrant_100k_tuned_results, 1m-qdrant-service-streaming-production_streaming_load_qdrant_1m_tuned_results, 1m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_1m_results, 100k-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_100k_results, 10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results, 10m-qdrant-service-streaming-production_streaming_load_qdrant_10m_results`.
- Measured frontier: `sub_100k-wavemind-numpy-streaming-production_streaming_load_smoke_results, 100k-qdrant-service-production_load_qdrant_100k_tuned_results, 1m-qdrant-service-streaming-production_streaming_load_qdrant_1m_tuned_results, 1m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_1m_results, 100k-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_100k_results, 10m-qdrant-sharded-service-streaming-production_streaming_load_qdrant_sharded_10m_results, 10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results, 10m-qdrant-service-streaming-production_streaming_load_qdrant_10m_results`.
- Planned frontier: `qdrant-sharded-100m, faiss-ivfpq-50m`.

## Measured Cost Frontier
Expand All @@ -27,8 +27,8 @@ Measured rows come from checked-in load artifacts. Planned rows are capacity and
| 8 | 1m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_1m_results | 1m | WaveMind faiss-ivfpq-persisted streaming | 1,000,000 | 0.99 | 4.992 | pass | $0.694 | $182.74 | `benchmarks/production_streaming_load_ivfpq_1m_results.json` |
| 9 | 100k-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_100k_results | 100k | WaveMind faiss-ivfpq-persisted streaming | 100,000 | 0.96 | 1.104 | pass | $0.694 | $182.52 | `benchmarks/production_streaming_load_ivfpq_100k_results.json` |
| 10 | 1m-wavemind-faiss-persisted-production_load_faiss_1m_results | 1m | WaveMind faiss-persisted | 1,000,000 | 1 | 57.71 | scale_required | $4.167 | $1,095.24 | `benchmarks/production_load_faiss_1m_results.json` |
| 11 | 10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results | 10m | WaveMind faiss-ivfpq-persisted streaming | 10,000,000 | 0.99 | 60.13 | scale_required | $4.861 | $1,279.88 | `benchmarks/production_streaming_load_ivfpq_10m_results.json` |
| 12 | 10m-qdrant-service-streaming-production_streaming_load_qdrant_10m_results | 10m | Qdrant service streaming | 10,000,000 | 0.975 | 43.27 | scale_required | $3.472 | $914.88 | `benchmarks/production_streaming_load_qdrant_10m_results.json` |
| 11 | 10m-qdrant-sharded-service-streaming-production_streaming_load_qdrant_sharded_10m_results | 10m | Qdrant sharded service streaming | 10,000,000 | 0.993 | 71.28 | scale_required | $4.722 | $3,104.88 | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` |
| 12 | 10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results | 10m | WaveMind faiss-ivfpq-persisted streaming | 10,000,000 | 0.99 | 60.13 | scale_required | $4.861 | $1,279.88 | `benchmarks/production_streaming_load_ivfpq_10m_results.json` |

## Planned Cost Frontier

Expand Down
6 changes: 3 additions & 3 deletions benchmarks/PRODUCTION_EVIDENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ multi-region, managed-serverless, 50M, or 100M production scale cannot.
| metric | value |
|---|---:|
| overall status | `action_required` |
| passed requirements | `3` |
| action required | `5` |
| passed requirements | `4` |
| action required | `4` |
| failed requirements | `0` |
| total requirements | `8` |

Expand All @@ -18,7 +18,7 @@ multi-region, managed-serverless, 50M, or 100M production scale cannot.
| External HTTP active-active regions | `action_required` | no checked-in external HTTP active-active region result; issues: missing artifact | `benchmarks/external_http_active_active_results.json` | `gh workflow run external-http-active-active.yml -f regions="us-east=https://wm-us.example.com,eu-west=https://wm-eu.example.com,ap-south=https://wm-ap.example.com" -f namespace_count=16 -f p99_slo_ms=1500 -f fail_on_slo=true -f commit_results=true` | Remote multi-region active-active memory convergence. |
| Managed/serverless remote telemetry | `action_required` | missing remote serverless telemetry; issues: missing artifact | `deploy/serverless/observed-telemetry.remote.json` | `gh workflow run serverless-observed-telemetry.yml -f nodes="https://wm-a.example.com,https://wm-b.example.com" -f seed_mode=first -f commit_results=true` | Hosted/serverless p99, cold-start, error-rate, and scale-out SLO. |
| 10M Qdrant service load | `pass` | Qdrant service streaming: vectors 10000000, recall 0.975, p99 43.266599997878075 ms, cost valid_slo, source be2cebfd776b, run local-qdrant10m-quantized-ef1024-v1182-20260711 | `benchmarks/production_streaming_load_qdrant_10m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 100.0 --replicas 3 --autoscaling-max-replicas 24 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_10m_results.json --checkpoint-path state/production-runs/qdrant-service-10000000.checkpoint.json` | 10M Qdrant service-backed candidate index SLO. |
| 10M sharded Qdrant service load | `action_required` | no checked-in 10,000,000-vector result for Qdrant sharded service streaming; issues: missing artifact | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json` | Horizontally sharded Qdrant service recall/latency SLO. |
| 10M sharded Qdrant service load | `pass` | Qdrant sharded service streaming: vectors 10000000, recall 0.9925, p99 71.27849999233149 ms, cost valid_slo, source 11d0be8c62e8, run local-qdrant-sharded10m-4x-v1182-20260711 | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json` | Horizontally sharded Qdrant service recall/latency SLO. |
| 10M pgvector service load | `action_required` | no checked-in 10,000,000-vector result for WaveMind pgvector streaming; issues: missing artifact | `benchmarks/production_streaming_load_pgvector_10m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines pgvector-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 100.0 --replicas 3 --autoscaling-max-replicas 24 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_pgvector_10m_results.json --checkpoint-path state/production-runs/pgvector-service-10000000.checkpoint.json` | 10M PostgreSQL/pgvector service candidate-index SLO. |
| 50M FAISS IVF-PQ streaming load | `pass` | WaveMind faiss-ivfpq-persisted streaming: vectors 50000000, recall 0.9705, p99 73.10794899996154 ms, cost valid_slo, source a9ea9d7f8027, run github-actions-29067655616 | `benchmarks/production_streaming_load_ivfpq_50m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 50000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 1000000 --engines faiss-ivfpq-persisted --target-recall 0.95 --target-p99-ms 100.0 --target-qps 100.0 --replicas 3 --autoscaling-max-replicas 24 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_ivfpq_50m_results.json --checkpoint-path state/production-runs/faiss-ivfpq-persisted-50000000.checkpoint.json` | 50M compressed local/persistent FAISS profile. |
| 100M remote load result | `action_required` | no checked-in 100,000,000-vector result for Qdrant sharded service streaming; issues: missing artifact | `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 100000000 --dim 128 --queries 5000 --top-k 10 --seed 42 --noise 0.08 --batch-size 10000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 500.0 --replicas 8 --autoscaling-max-replicas 128 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_100m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-100000000.checkpoint.json` | 100M+ memories with measured sharded-Qdrant recall, p99, and cost SLO. |
5 changes: 2 additions & 3 deletions benchmarks/PRODUCTION_EVIDENCE_BUNDLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ claim boundaries, and the exact next actions required to unlock blocked claims.
| metric | value |
|---|---:|
| claim status | `claims_limited` |
| strict evidence | `3/8` |
| strict evidence | `4/8` |
| preflight ready | `0/8` |
| production readiness | `pass` |
| readiness score | `1.0` |
Expand All @@ -16,7 +16,7 @@ claim boundaries, and the exact next actions required to unlock blocked claims.
| production scale run contract | `available` |
| production scale profiles | `5` |
| production scale target memories | `180000000` |
| next actions | `5` |
| next actions | `4` |

## Claim Boundaries

Expand Down Expand Up @@ -44,6 +44,5 @@ claim boundaries, and the exact next actions required to unlock blocked claims.
|---|---|---|---|---|---|
| External HTTP active-active regions | `action_required` | `action_required` | `benchmarks/external_http_active_active_results.json` | `WAVEMIND_ACTIVE_ACTIVE_REGIONS, WAVEMIND_ACTIVE_ACTIVE_REGIONS_MANIFEST_JSON; issues: missing artifact` | `gh workflow run external-http-active-active.yml -f regions="$WAVEMIND_ACTIVE_ACTIVE_REGIONS" -f commit_results=true` |
| Managed/serverless remote telemetry | `action_required` | `action_required` | `deploy/serverless/observed-telemetry.remote.json` | `WAVEMIND_SERVERLESS_NODES; issues: missing artifact` | `gh workflow run serverless-observed-telemetry.yml -f nodes="$WAVEMIND_SERVERLESS_NODES" -f seed_mode=first -f commit_results=true` |
| 10M sharded Qdrant service load | `action_required` | `action_required` | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS; issues: missing artifact` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json` |
| 10M pgvector service load | `action_required` | `action_required` | `benchmarks/production_streaming_load_pgvector_10m_results.json` | `WAVEMIND_PGVECTOR_DSN; issues: missing artifact` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines pgvector-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 100.0 --replicas 3 --autoscaling-max-replicas 24 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_pgvector_10m_results.json --checkpoint-path state/production-runs/pgvector-service-10000000.checkpoint.json` |
| 100M remote load result | `action_required` | `action_required` | `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json` | `WAVEMIND_QDRANT_URLS; issues: missing artifact` | `python benchmarks/production_streaming_load_benchmark.py --sizes 100000000 --dim 128 --queries 5000 --top-k 10 --seed 42 --noise 0.08 --batch-size 10000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 500.0 --replicas 8 --autoscaling-max-replicas 128 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_100m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-100000000.checkpoint.json` |
6 changes: 3 additions & 3 deletions benchmarks/PRODUCTION_EVIDENCE_DISPATCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ strict production-evidence validation.
|---|---:|
| overall status | `action_required` |
| ready to dispatch | `0` |
| blocked by preflight | `5` |
| complete | `3` |
| blocked by preflight | `4` |
| complete | `4` |
| total jobs | `8` |
| runner label | `self-hosted-large` |
| commit results default | `False` |
Expand All @@ -23,7 +23,7 @@ strict production-evidence validation.
| External HTTP active-active regions | `blocked_by_preflight` | `remote-service` | `external-http-active-active.yml` | `benchmarks/external_http_active_active_results.json` | `WAVEMIND_ACTIVE_ACTIVE_REGIONS, WAVEMIND_ACTIVE_ACTIVE_REGIONS_MANIFEST_JSON` |
| Managed/serverless remote telemetry | `blocked_by_preflight` | `remote-service` | `serverless-observed-telemetry.yml` | `deploy/serverless/observed-telemetry.remote.json` | `WAVEMIND_SERVERLESS_NODES` |
| 10M Qdrant service load | `complete` | `service-scale-10m` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_qdrant_10m_results.json` | `WAVEMIND_QDRANT_URL` |
| 10M sharded Qdrant service load | `blocked_by_preflight` | `service-scale-10m` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS` |
| 10M sharded Qdrant service load | `complete` | `service-scale-10m` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS` |
| 10M pgvector service load | `blocked_by_preflight` | `service-scale-10m` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_pgvector_10m_results.json` | `WAVEMIND_PGVECTOR_DSN` |
| 50M FAISS IVF-PQ streaming load | `complete` | `large-local-index` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_ivfpq_50m_results.json` | `WAVEMIND_FAISS_IVFPQ_PATH` |
| 100M remote load result | `blocked_by_preflight` | `hundred-million-service` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json` | `WAVEMIND_QDRANT_URLS` |
Expand Down
Loading
Loading