diff --git a/README.md b/README.md index cf8ed36..f9b542d 100644 --- a/README.md +++ b/README.md @@ -2075,8 +2075,8 @@ remote/large-N production claims are unlocked. Deployment admission: `wavemind production-admission --target-memories 100000000 --engine qdrant-sharded-service --fail-on-blocked` is the final deploy-facing check; it keeps a requested production profile blocked until its matching strict evidence -artifact passes. Persisted 50M FAISS and single-service 10M Qdrant are admitted; -the 10M sharded Qdrant, 10M pgvector, and 100M sharded profiles remain blocked. +artifact passes. Persisted 50M FAISS, single-service 10M Qdrant, and four-service +10M sharded Qdrant are admitted; 10M pgvector and 100M sharded Qdrant remain blocked. `--allow-plan-only` reports the next run contract without admitting production. The same gate can protect the API process itself: `wavemind serve --require-production-admission --production-target-memories @@ -2138,13 +2138,13 @@ public claim boundaries stable: | Claim area | Current public status | Source of truth | Not proven yet | |---|---|---|---| | Production readiness | WaveMind core readiness is gated by checked-in artifacts before release. | `benchmarks/production_readiness_results.json`, `benchmarks/PRODUCTION_READINESS.md` | Missing external competitor credentials should not be treated as WaveMind core failure, but they still limit competitor claims. | -| Strict production evidence | The gate now passes `3/8` requirements: persisted 50M FAISS reaches recall@10 `0.9705` and p99 `73.11 ms`; single-service 10M Qdrant reaches recall@10 `0.975` and p99 `43.27 ms`; the non-loopback Kubernetes cluster passes success/failover `1.00`, query p99 `84.80 ms`, batch p99 `148.82 ms`, and physical-worker attestation `10/10`. | `benchmarks/production_evidence_results.json`, `benchmarks/PRODUCTION_EVIDENCE.md`, `benchmarks/production_evidence_gate.py`, `wavemind production-evidence --strict` | Five requirements remain: remote active-active, managed serverless telemetry, 10M sharded Qdrant, 10M pgvector, and the 100M sharded service run. | +| Strict production evidence | The gate now passes `4/8` requirements: persisted 50M FAISS reaches recall@10 `0.9705` and p99 `73.11 ms`; single-service 10M Qdrant reaches `0.975` and `43.27 ms`; four-service 10M sharded Qdrant reaches `0.9925` and `71.28 ms`; the non-loopback Kubernetes cluster passes success/failover `1.00`, query p99 `84.80 ms`, batch p99 `148.82 ms`, and physical-worker attestation `10/10`. | `benchmarks/production_evidence_results.json`, `benchmarks/PRODUCTION_EVIDENCE.md`, `benchmarks/production_evidence_gate.py`, `wavemind production-evidence --strict` | Four requirements remain: remote active-active, managed serverless telemetry, 10M pgvector, and the 100M sharded service run. | | Production evidence preflight | Remote endpoint/env/path prerequisites are checked before launching expensive strict-evidence jobs. | `benchmarks/production_evidence_preflight_results.json`, `benchmarks/PRODUCTION_EVIDENCE_PREFLIGHT.md`, `wavemind production-evidence-preflight --write-artifacts` | A ready preflight is not a passing evidence result; it only proves the environment is ready to run the remote/large-N jobs. | | Production evidence env contract | Secret-safe operator map from every strict-evidence env var to the workflows, claims, artifacts, GitHub Actions secrets, input bindings, and `.env.example` placeholders it unlocks. | `benchmarks/production_evidence_env_contract.json`, `benchmarks/PRODUCTION_EVIDENCE_ENV.md`, `deploy/cluster/production-evidence.env.example`, `wavemind production-evidence-env --write-artifacts` | It does not unlock production claims; it prevents ambiguous or unsafe production evidence launches and keeps secrets out of checked-in artifacts. | | Production evidence dispatch | Secret-safe workflow dispatch contract for every unfinished strict-evidence job, including safe `commit_results=false` launch commands, publish commands, required env/secrets, and artifact promotion commands. | `benchmarks/production_evidence_dispatch_results.json`, `benchmarks/PRODUCTION_EVIDENCE_DISPATCH.md`, `wavemind production-evidence-dispatch --write-artifacts` | A dispatch plan only launches or reviews evidence runs; it does not unlock production claims until downloaded artifacts pass ingest and strict validation. | -| Strict evidence readiness | Operator runbook that joins strict evidence, preflight, dispatch, scale plans, scale gaps, release claims, and freshness into one table of blockers, locked claims, safe dispatch commands, ingest commands, and validation commands. | `benchmarks/strict_evidence_readiness_results.json`, `benchmarks/STRICT_EVIDENCE_READINESS.md`, `python benchmarks/strict_evidence_readiness_report.py` | Current readiness is `action_required`: non-loopback Kubernetes cluster evidence, 50M FAISS, and single-service 10M Qdrant are complete; the remaining five remote/service jobs are mapped but not yet proven. | +| Strict evidence readiness | Operator runbook that joins strict evidence, preflight, dispatch, scale plans, scale gaps, release claims, and freshness into one table of blockers, locked claims, safe dispatch commands, ingest commands, and validation commands. | `benchmarks/strict_evidence_readiness_results.json`, `benchmarks/STRICT_EVIDENCE_READINESS.md`, `python benchmarks/strict_evidence_readiness_report.py` | Current readiness is `action_required`: non-loopback Kubernetes cluster evidence, 50M FAISS, single-service 10M Qdrant, and four-service 10M sharded Qdrant are complete; four remote/service jobs remain. | | Production evidence bundle | Single operator-facing status contract that combines strict gate, preflight, readiness, artifact audit, claim boundaries, next actions, and release exit behavior. | `benchmarks/production_evidence_bundle_results.json`, `benchmarks/PRODUCTION_EVIDENCE_BUNDLE.md`, `wavemind production-evidence-bundle --write-artifacts` | `claims_limited` is expected until the strict remote/large-N artifacts pass. | -| Release claims | Compact release-facing claim contract for GitHub Releases and launch posts: what is safe to claim, what remains locked, and which command unlocks the next evidence tier. | `benchmarks/release_claims_results.json`, `benchmarks/RELEASE_CLAIMS.md`, `wavemind release-claims --write-artifacts --fail-on-blocked` | `core_release_ready`, non-loopback Kubernetes cluster SLO, 50M persisted FAISS, and single-service 10M Qdrant are supported; remote multi-region, sharded 10M/100M, and 10M pgvector claims remain locked. | +| Release claims | Compact release-facing claim contract for GitHub Releases and launch posts: what is safe to claim, what remains locked, and which command unlocks the next evidence tier. | `benchmarks/release_claims_results.json`, `benchmarks/RELEASE_CLAIMS.md`, `wavemind release-claims --write-artifacts --fail-on-blocked` | `core_release_ready`, non-loopback Kubernetes cluster SLO, 50M persisted FAISS, single-service 10M Qdrant, and sharded 10M Qdrant are supported; remote multi-region, 100M sharded, and 10M pgvector claims remain locked. | | Agent impact leaderboard | Behavioral benchmark evidence is aggregated across agent coherence, dynamic-memory policy, long-memory retrieval, and LongMemEval answer quality. | `benchmarks/agent_impact_results.json`, `benchmarks/AGENT_IMPACT.md`, `benchmarks/agent_impact_leaderboard.py` | It proves lift on the listed checked-in scenarios only; it does not claim general agent success outside those tasks. | | Memory OS intelligence | Adaptive-worker evidence is aggregated across scale readiness, agent coherence, staging canary, and admission artifacts. It tracks hot-query prewarm, transition-learned predictive prefetch, priority learning, adaptive forgetting, concept consolidation, Redis coordination, canary status, and production-admission boundaries. | `benchmarks/memory_os_intelligence_results.json`, `benchmarks/MEMORY_OS_INTELLIGENCE.md`, `benchmarks/memory_os_intelligence_report.py` | It proves Memory OS behavior on checked-in fixtures; unattended production automation remains locked until real shared Redis, distributed lock, runtime env, and large-scale evidence pass. | | Cluster autoscale | Cluster/operator evidence is pulled into a dedicated public report. It tracks deterministic shard placement, node/zone loss availability, autoscale planning, rebalance checkpoints, Kubernetes operator reconciliation, quorum safety, HTTP sharding, active-active convergence, CRDT field state, and the 100M capacity envelope. | `benchmarks/cluster_autoscale_results.json`, `benchmarks/CLUSTER_AUTOSCALE.md`, `benchmarks/cluster_autoscale_report.py` | It is a deterministic capacity and operator evidence report, not a real 100M vector-query latency benchmark or managed Kubernetes production SLO. | @@ -2152,8 +2152,8 @@ public claim boundaries stable: | Kubernetes active-active region failure | Three PVC-backed replicated regions in three worker zones converge `48` initial writes, continue `32` writes plus a delete while region B is physically unavailable, then recover at `1.00` convergence and delete suppression with an idempotent final sync. | `benchmarks/kubernetes_active_active_region_smoke_results.json`, `benchmarks/kubernetes_active_active_region_smoke.py`, `wavemind active-active-drill` | This proves the active-active protocol across non-loopback Kubernetes services and a physical zone outage in ephemeral CI. Independent remote regions are still required for strict active-active admission. | | Kubernetes serverless lifecycle | PVC-backed PostgreSQL, Qdrant, and Redis preserve `24/24` memories through two scale-to-zero cycles; three zone-spread workers achieve write/delete coherence at `3/3` within `1.14 s`, and burst p99 remains below `2 s`. | `benchmarks/kubernetes_serverless_lifecycle_smoke_results.json`, `benchmarks/kubernetes_serverless_lifecycle_smoke.py`, `.github/workflows/kubernetes-operator-smoke.yml` | This proves external-state lifecycle and bounded worker-cache convergence in ephemeral non-loopback Kubernetes. Managed Knative/KEDA endpoints and remote telemetry are still required for strict serverless admission. | | Kubernetes PostgreSQL/Qdrant DR | A checksummed PostgreSQL backup restores into an independent namespace with fresh PVCs and an empty Qdrant service; recall and index parity are `24/24`, including after recovery API replacement. | `benchmarks/kubernetes_postgres_qdrant_dr_smoke_results.json`, `benchmarks/kubernetes_postgres_qdrant_dr_smoke.py`, `.github/workflows/kubernetes-operator-smoke.yml` | This proves logical backup/restore and vector-index reconstruction in ephemeral Kubernetes. It is not managed PostgreSQL PITR, remote object-store recovery, or multi-region DR. | -| Scale gap matrix | Large-N proof status for 10M Qdrant, 10M sharded Qdrant, 10M pgvector, 50M FAISS IVF-PQ, and 100M sharded Qdrant. It joins strict evidence, preflight, run commands, missing env, and measured baselines. | `benchmarks/scale_gap_results.json`, `benchmarks/SCALE_GAP.md`, `wavemind scale-gap --write-artifacts` | `2/5` profiles are complete: 50M persisted FAISS and single-service 10M Qdrant; 10M sharded Qdrant, 10M pgvector, and 100M sharded Qdrant remain. | -| Cost-efficiency leaderboard | Cost, latency, recall, SLO, and memory-count evidence are ranked across measured production-load artifacts and the remaining plan-only 10M-service/100M contracts. | `benchmarks/cost_efficiency_results.json`, `benchmarks/COST_EFFICIENCY.md`, `benchmarks/cost_efficiency_leaderboard.py` | The measured 50M FAISS and 10M Qdrant rows are evidence; remaining planned rows are capacity/cost contracts only. | +| Scale gap matrix | Large-N proof status for 10M Qdrant, 10M sharded Qdrant, 10M pgvector, 50M FAISS IVF-PQ, and 100M sharded Qdrant. It joins strict evidence, preflight, run commands, missing env, and measured baselines. | `benchmarks/scale_gap_results.json`, `benchmarks/SCALE_GAP.md`, `wavemind scale-gap --write-artifacts` | `3/5` profiles are complete: 50M persisted FAISS, single-service 10M Qdrant, and four-service 10M sharded Qdrant; 10M pgvector and 100M sharded Qdrant remain. | +| Cost-efficiency leaderboard | Cost, latency, recall, SLO, and memory-count evidence are ranked across measured production-load artifacts and the remaining plan-only 10M-service/100M contracts. | `benchmarks/cost_efficiency_results.json`, `benchmarks/COST_EFFICIENCY.md`, `benchmarks/cost_efficiency_leaderboard.py` | The measured 50M FAISS, 10M Qdrant, and 10M sharded Qdrant rows are evidence; remaining planned rows are capacity/cost contracts only. | | Production admission | Deployment-facing gate for a requested memory count and engine. It maps the requested 10M/50M/100M deployment to the required strict evidence profile and fails deploys until that artifact passes. | `benchmarks/production_admission_results.json`, `benchmarks/PRODUCTION_ADMISSION.md`, `wavemind production-admission --target-memories 100000000 --engine qdrant-sharded-service --fail-on-blocked` | Current 100M sharded Qdrant status is `plan_only`, not admitted: the run contract exists, but `production_streaming_load_qdrant_sharded_100m_results.json` is still missing. | | Cluster admission | Deployment-facing gate for non-loopback service-node rollouts. It requires strict load evidence, a ready preflight, and an exact node ID-to-URL match for the requested target. | `benchmarks/cluster_admission_results.json`, `benchmarks/CLUSTER_ADMISSION.md`, `wavemind cluster-admission --fail-on-blocked --write-artifacts` | The attested kind target is `admitted`. A different staging or production target remains blocked until it produces matching endpoint-specific evidence. | | Active-active admission | Deployment-facing gate for remote multi-region active-active rollout. It admits only when the strict external HTTP active-active artifact passes; local/loopback runs remain development evidence. | `benchmarks/active_active_admission_results.json`, `benchmarks/ACTIVE_ACTIVE_ADMISSION.md`, `wavemind active-active-admission --allow-plan-only --write-artifacts` | Current status is `plan_only`, not admitted: `benchmarks/external_http_active_active_results.json` is still missing and remote region env is not configured. | @@ -2165,10 +2165,10 @@ public claim boundaries stable: | Memory OS policy bundle | Operator-facing runtime policy manifest generated from canary, policy-evolution, and admission artifacts. It emits enabled task ids, required Redis/lock env, observability metrics, Kubernetes/CronJob patch data, and promotion gates. | `benchmarks/memory_os_policy_bundle_results.json`, `benchmarks/MEMORY_OS_POLICY_BUNDLE.md`, `wavemind memory-os-policy-bundle --write-artifacts` | Current status is `staging_ready`: it can promote checked Memory OS behavior to staging, but production remains locked while `memory-os-admission` is `plan_only`. | | Memory OS admission | Deployment-facing gate for adaptive workers. It checks scheduler safety, hot-query audit signal, Redis cache wiring, distributed lock wiring, singleton/idempotent mutations, policy coverage, and strict architecture boundaries before Memory OS workers become production automation. | `benchmarks/memory_os_admission_results.json`, `benchmarks/MEMORY_OS_ADMISSION.md`, `wavemind memory-os-admission --target-memories 10000000 --namespace-count 4096 --deployment production --allow-plan-only --write-artifacts` | Current 10M Memory OS status is `plan_only`, not admitted: the worker plan exists, but staging query-audit traffic, Redis/lock runtime env, and strict million-plus architecture evidence are still required. | | Production scale run planner | One command plans the next large-N jobs across 10M Qdrant, 10M sharded Qdrant, 10M pgvector, 50M FAISS IVF-PQ, and 100M sharded Qdrant, including env, checkpoint, storage, SLO, monthly budget, cost per 1M memories, compute cost per 1M queries, plan-only Pareto frontier, and output artifact contracts. | `benchmarks/production_scale_run_plan.json`, `wavemind production-scale-plan --write-artifact` | This is a run contract and preflight only; it does not replace the real latency/recall result artifacts. | -| 10M memory-scale profile | Checked-in compressed FAISS IVF-PQ and real single-service Qdrant streaming profiles are reported in the generated leaderboard. Qdrant 1.18.2 over a POSIX Docker named volume with gRPC, original vectors on disk, and INT8 quantization reaches recall@10 `0.975` and p99 `43.27 ms` across `2,000` queries. | `benchmarks/production_streaming_load_ivfpq_10m_results.json`, `benchmarks/production_streaming_load_qdrant_10m_results.json`, `benchmarks/production_streaming_load_qdrant_10m_plan.json` | The 10M sharded Qdrant and pgvector service comparisons remain unfinished. | +| 10M memory-scale profile | Checked-in compressed FAISS IVF-PQ, real single-service Qdrant, and real four-service sharded Qdrant profiles are reported in the generated leaderboard. Single Qdrant reaches recall@10 `0.975`, p99 `43.27 ms`; sharded Qdrant reaches `0.9925`, `71.28 ms` across `2,000` queries with exact 2.5M-per-shard balance. | `benchmarks/production_streaming_load_ivfpq_10m_results.json`, `benchmarks/production_streaming_load_qdrant_10m_results.json`, `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | The 10M pgvector service comparison remains unfinished. | | 50M persisted FAISS IVF-PQ | Real GitHub-hosted run over `50,000,000` 128D vectors and `2,000` queries. Adaptive `nprobe` selected `512`: recall@10 `0.9705`, p99 `73.11 ms`, valid cost/SLO evidence. | `benchmarks/production_streaming_load_ivfpq_50m_results.json`, `benchmarks/production_streaming_load_50m_plan.json` | This proves a compressed single-node persisted index, not a 100M distributed service cluster. | | pgvector tuning | Real PostgreSQL/pgvector service profile now separates baseline HNSW, exact recall floor, and iterative HNSW tuning. | `benchmarks/production_pgvector_tuning_results.json` | This is a 50k service-backed tuning profile, not yet the 100k/1M production load SLO artifact. | -| Qdrant streaming | Real Qdrant streaming smoke exists, a tuned 1M service run passes SLO, the strict single-service 10M run reaches recall@10 `0.975` and p99 `43.27 ms`, a real two-service sharded smoke passes, and horizontally sharded 10M/100M contracts are checked in. | `benchmarks/production_streaming_load_qdrant_smoke_results.json`, `benchmarks/production_streaming_load_qdrant_1m_results.json`, `benchmarks/production_streaming_load_qdrant_1m_tuned_results.json`, `benchmarks/production_streaming_load_qdrant_10m_results.json`, `benchmarks/production_streaming_load_qdrant_sharded_smoke_results.json`, `benchmarks/production_streaming_load_qdrant_sharded_10m_plan.json`, `benchmarks/production_streaming_load_qdrant_sharded_100m_plan.json` | The sharded 10M/100M Qdrant results are not claimed until the matching result artifacts are produced by real runs. | +| Qdrant streaming | Real Qdrant streaming smoke exists, tuned 1M and strict single-service 10M runs pass quality/latency gates, and the real four-service sharded 10M run reaches recall@10 `0.9925`, p99 `71.28 ms`. | `benchmarks/production_streaming_load_qdrant_smoke_results.json`, `benchmarks/production_streaming_load_qdrant_1m_results.json`, `benchmarks/production_streaming_load_qdrant_1m_tuned_results.json`, `benchmarks/production_streaming_load_qdrant_sharded_smoke_results.json`, `benchmarks/production_streaming_load_qdrant_10m_plan.json`, `benchmarks/production_streaming_load_qdrant_10m_results.json`, `benchmarks/production_streaming_load_qdrant_sharded_10m_plan.json`, `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json`, `benchmarks/production_streaming_load_qdrant_sharded_100m_plan.json` | The 100M sharded Qdrant result remains locked until its real run artifact is produced. | | pgvector streaming | Real PostgreSQL/pgvector streaming smoke exists, and a 10M plan-only service contract is checked in. | `benchmarks/production_streaming_load_pgvector_smoke_results.json`, `benchmarks/production_streaming_load_pgvector_10m_plan.json` | The 10M pgvector service result is not claimed until `production_streaming_load_pgvector_10m_results.json` is produced by a real run. | | HTTP cluster load | The checked artifact runs the mixed workload from inside Kubernetes against four pod-DNS API nodes: success/failover/delete suppression `1.00`, repaired replica `1`, query p99 `84.80 ms`, batch p99 `148.82 ms`, and external batch requests `24 -> 1`. Bulk lifecycle batch p99 is reported separately at `6694.76 ms`. | `benchmarks/http_cluster_load_results.json`, `.github/workflows/kubernetes-operator-smoke.yml`, `.github/workflows/external-http-cluster-load.yml` | This is ephemeral non-loopback Kubernetes evidence, not managed multi-region or million-scale service evidence. | | HTTP active-active regions | Local multi-process API-region evidence exists, and the external URL-based contract now has a loopback artifact. The external workflow can run the same namespace-delta contract against real regional API URLs. | `benchmarks/local_http_active_active_smoke_results.json`, `benchmarks/external_http_active_active_loopback_results.json`, `.github/workflows/external-http-active-active.yml` | Local/loopback active-active evidence is not a remote Kubernetes/serverless multi-region result until `benchmarks/external_http_active_active_results.json` is produced by a real run. | @@ -2223,7 +2223,7 @@ Current read: | ANN/index curve | At 50000 generated 128-d vectors, NumPy exact keeps `recall@10 1.000` at `1.99 ms`; quantized int8 keeps `0.934` at `16.27 ms`; Annoy is faster at `3.21 ms` but drops to `0.730` recall; FAISS flat keeps `1.000` recall with a cold-start-inflated `80.47 ms` average and `2.42 ms` p95; Qdrant local keeps `1.000` recall at `33.82 ms`. | Current local scale boundary is clear: top-k selection is faster, quantized is memory-oriented but still not a latency win, Annoy needs recall tuning, and service-mode indexes are the production path. | | Production load | At 100000 generated 128-d vectors, service-mode Qdrant reaches `recall@10 1.000`, avg `10.28 ms`, p99 `21.26 ms`, passes the checked-in production SLO gate (`recall >= 0.95`, `p99 <= 100 ms`, `100 qps`, 3 replicas, HPA max 24), and estimates `$1.39` per 1M queries with `$365.02` monthly target cost. At 1M over 100 queries, persisted FAISS reaches `recall@10 1.000`, avg `39.12 ms`, p99 `57.71 ms`, and estimates `$4.17` per 1M queries with 6 replicas for 100 qps. The older 1M Qdrant tuned production-load profile reaches `recall@10 0.984`, avg `82.57 ms`, p99 `137.86 ms`; the newer streaming 1M Qdrant profile closes that p99 gap after safe upsert chunking, wait-after-build, and query warmup. | 100k Qdrant, 1M persisted FAISS, and the tuned 1M Qdrant streaming profile now pass recall/p99 production gates on the tested machine. The older Qdrant load profile stays checked in as evidence that cold/untuned service tails can fail. | | pgvector tuning | On a real PostgreSQL/pgvector service at 50000 vectors, baseline HNSW reaches `recall@10 0.834`, exact mode reaches `1.000` with p99 `76.98 ms`, and iterative HNSW reaches `0.970` with p99 `55.19 ms`. Qdrant service remains the speed reference at `1.000` recall and p99 `17.84 ms`. | pgvector now has a measured production tuning path. Exact mode proves correctness; iterative scan passes the 50k recall/p99 gate and should be promoted into 100k/1M load profiles next. | -| Streaming production load | `benchmarks/production_streaming_load_benchmark.py` generates and inserts vectors in batches, stores only query source vectors outside the index, and measures target-recall, p99, SLO, and cost. The checked-in 10M compressed FAISS IVF-PQ run reaches recall@10 `0.990`, p99 `60.13 ms`; the real 50M persisted FAISS run reaches recall@10 `0.9705`, p99 `73.11 ms`; tuned 1M Qdrant reaches recall@10 `1.000`, p99 `26.37 ms`; and strict 10M Qdrant reaches recall@10 `0.975`, p99 `43.27 ms` over `2,000` queries. | The FAISS results are compressed target-recall profiles. The unfinished strict profiles are 10M sharded Qdrant, 10M pgvector, and 100M sharded Qdrant. | +| Streaming production load | `benchmarks/production_streaming_load_benchmark.py` generates and inserts vectors in bounded batches, stores only query source vectors outside the index, and measures target-recall, p99, SLO, and cost. 10M compressed FAISS reaches recall@10 `0.990`, p99 `60.13 ms`; 50M persisted FAISS reaches `0.9705`, `73.11 ms`; strict 10M Qdrant reaches `0.975`, `43.27 ms`; and strict four-service 10M sharded Qdrant reaches `0.9925`, `71.28 ms`. | The FAISS results are compressed target-recall profiles. The unfinished strict profiles are 10M pgvector and 100M sharded Qdrant. | | Structured memory report | Dedicated status view for image/audio/video/3D/table/event/graph payloads. It reports `7` modalities, structured precision@1 `1.000`, cross-modal precision@1 `1.000`, persisted vector rate `1.000`, provenance `1.000`, precomputed-vector precision@1 `1.000`, temporal event precision@1 `1.000`, knowledge-graph precision@1 `1.000`, graph path precision@1 `1.000`, and all gate checks passing. | This makes the multimodal/structured roadmap evidence visible outside the larger scale-readiness row. It is still a deterministic fixture and external-vector contract, not a claim of broad production multimodal model quality. | | Scale readiness | Deterministic 1M-memory simulation validates 4096 namespace placements over 4 nodes with replication factor 2, node-loss availability `1.000`, zone-loss availability `1.000`, Kubernetes `StatefulSet`, `HorizontalPodAutoscaler`, repair `CronJob`, Memory OS `CronJob`, majority control-plane lease/config revision safety with stale-leader, stale-revision, and minority-commit rejection, operator-style `WaveMindCluster` reconciliation for `4096` namespaces, operator status phase `Ready` with resources/capacity/autoscaling/repair/Memory OS/production-admission/control-plane conditions true, hot-cache hit rate `0.920`, query-audit prewarm warmed `1` query with prewarm hit `true`, query-vector cache local hit rate `0.995` with `1` encode call, Redis query-vector cache shared across workers `true`, FastAPI `/query/batch` answers 100 recall queries with 1 HTTP request and batch hit rate `0.990`, shared Redis rate limiter allows `4` and limits `1` across 2 workers, Redis-compatible shared cache visible across workers, Memory OS Redis prewarm warmed `2` queries, predictive prefetch warmed `6` queries, transition-prefetch hit `true` on `budget recall -> risk limits`, explicit feedback events `8`, Redis Memory OS demoted cold memories, Redis cross-worker hit `true`, Redis namespace invalidation `true`, Redis Memory OS architecture advice `architecture_required`, API cache invalidation on `/remember`, `/feedback`, `/feedback/batch`, and `/forget` prevents stale cached recall, batch feedback accepts `2`, rejects `1`, writes `2` audit events, and updates positive/negative priority, Memory OS found `2` hot queries, warmed `2`, predictively warmed `6`, demoted cold memories, purged `1` expired memory, created `1` durable concept, emitted a production execution plan with `safe_to_run=true`, Redis and lock environment requirements, singleton state-mutating tasks, and worker-pool `cache-prewarm`, and emitted production architecture advice for service-index, namespace-sharding, production-controls, replication-capacity, load-test, and multimodal readiness, service-mode distributed sharding recall after primary loss, service-mode repair copied `1` missing replica record with recall after repair `true`, real HTTP shard transport with proxy bypass `true`, HTTP repair copied `1` missing replica record, HTTP tombstone repair deleted `1` stale API record, concurrent HTTP writes `12`, concurrent query hit rate `1.000`, service-mode tombstone suppression before repair `true`, tombstone repair deleted `1` stale replica record, suppression after repair `true`, anti-entropy worker repaired `1` missing record and deleted `1` stale tombstone record, quorum-replicated runtime recall after node loss, missing-record repair, tombstone repair, concurrent runtime writes `12`, concurrent runtime query hit rate `1.000`, active-active namespace delta sync with cursor-based incremental record export and field-only hotness export, sustained active-active mesh sync across 3 independent regions and 3 namespaces with convergence `1.000`, delete suppression `1.000`, success `1.000`, 90 pair syncs, final no-op imports `0`, HTTP service-region active-active sync through FastAPI delta endpoints with convergence `1.000` and final no-op imports `0`, real local HTTP active-active service-region smoke with convergence `1.000`, delete suppression `1.000`, success `1.000`, and final no-op imports `0`, field-state CRDT convergence/idempotency/tombstone-wins, checksummed replicated snapshot/restore, offsite mirror verification, portable archive verification, S3-compatible upload/latest-metadata/download/retention verification, object-store DR drill `true`, SQLite recovery journal full restore `true`, point-in-time restore `true`, structured payload precision@1 `1.000`, cross-modal target-modality precision@1 `1.000`, cross-modal vector persistence `1.000`, precomputed external-vector precision@1 `1.000`, precomputed vector persistence `1.000`, cross-modal provenance rate `1.000`, temporal event precision@1 `1.000`, temporal persistence `1.000`, temporal provenance `1.000`, knowledge-graph direct/two-hop/three-hop/predicate precision `1/1/1/1`, graph path precision@1 `1.000`, graph persistence `1.000`, graph provenance `1.000`, and a deterministic 100M-memory capacity envelope with weighted rendezvous zone-aware placement, 128 nodes, 8 zones, RF=3, node/zone-loss availability `1.000`, distinct replica rate `1.000`, zone-spread rate `1.000`, valid capacity plan `true`, and a 128-to-160-node scale-out movement audit. | This proves routing, control-plane split-brain protection for config changes, Kubernetes deployment/operator/HPA/repair/Memory OS manifests and status conditions, production admission wiring for 10M+ targets, service-mode repair, real HTTP shard transport, service-boundary active-active delta sync, real multi-process active-active service-region sync, concurrent API safety for local WaveMind/SQLite nodes, concurrent replicated-runtime safety, tombstone-aware delete repair, anti-entropy background repair, explicit and batch recall feedback, query-vector cache, API batch recall, shared rate limiting, Memory OS adaptive prewarm/transition-learned predictive prefetch/consolidation/forgetting/architecture advice, Memory OS rollout safety contracts, local and Redis-compatible cache prewarm, shared cache invalidation and mutation safety, structured, temporal, cross-modal, and knowledge-graph payload retrieval, external precomputed-vector compatibility, distributed sharding, replicated-runtime, cursor-bounded namespace-delta sync, sustained active-active mesh convergence, distributed field-state convergence, offsite/archive/object-store backup lifecycle, SQLite point-in-time recovery, restore-drill foundations, and 100M-scale placement/capacity planning. | | Local HTTP cluster smoke | 4 real localhost API processes with isolated SQLite stores, RF=3, `read_fanout=1`, workers `4`: success `1.000`, failover hit `1.000`, delete suppression `1.000`, repaired replicas `1`, health `true`, degraded nodes `0`, p99 `348.83 ms`, SLO `true`. | This is the CI-friendly service-mode gate between in-process tests and remote external-node benchmarks. It catches HTTP transport, quorum, repair, delete-suppression, post-load node-health, and circuit-state regressions without needing external infrastructure. | @@ -2252,7 +2252,7 @@ by the production gate. | Production index profile | Docker-backed 50000-vector profile for persisted FAISS, Qdrant service, and PostgreSQL/pgvector HNSW. | implemented | FAISS / Qdrant service / pgvector | Keep service-mode candidate generation above `0.95` recall@10 and below 10 ms average query latency at 50000 vectors. | | Production load profile | 100k and 1M service-backed candidate-index checks with p95/p99 latency plus an explicit SLO/cost gate for recall, p99, QPS, replicas, HPA capacity, storage, monthly target cost, and cost per 1M queries. | implemented | Qdrant service / pgvector HNSW / FAISS persisted | Keep 100k Qdrant and 1M persisted FAISS green while tuning Qdrant/pgvector for the same 1M p99 gate. | | Qdrant 1M HNSW ef sweep | One 1M Qdrant collection queried with multiple `hnsw_ef` values and the same SLO gate. | implemented | Qdrant service | Keep the older sweep as a tail-latency regression baseline; the streaming 1M profile now proves the passing path with safe chunks, wait-after-build, and 100 warmup queries. | -| Production streaming load runner | Memory-bounded large-N runner that generates and inserts vectors in batches and measures target-recall, p99, SLO, and cost without storing the full corpus or exact-neighbor matrix in RAM. | implemented | FAISS persisted / FAISS IVF-PQ persisted / Qdrant service streaming / Qdrant sharded service streaming / pgvector streaming | Keep the passing 10M/50M compressed FAISS and strict 10M Qdrant profiles green, then complete 10M sharded Qdrant, 10M pgvector, and the real 100M distributed sharded-Qdrant run. | +| Production streaming load runner | Memory-bounded large-N runner that generates and inserts vectors in batches and measures target-recall, p99, SLO, and cost without storing the full corpus or exact-neighbor matrix in RAM. | implemented | FAISS persisted / FAISS IVF-PQ persisted / Qdrant service streaming / Qdrant sharded service streaming / pgvector streaming | Keep 10M/50M FAISS, strict 10M Qdrant, and strict four-service 10M sharded Qdrant green; next complete 10M pgvector and the real 100M distributed sharded-Qdrant run. | | Scale readiness profile | Cluster placement, node/zone-loss simulation, quorum report, control-plane majority lease/config revision safety, Kubernetes StatefulSet, HPA, repair CronJob, CRD status subresource, operator readiness/capacity/autoscaling/repair/production-admission/control-plane conditions, service-mode distributed namespace sharding, real HTTP shard transport, sustained mixed HTTP cluster load, replica repair, tombstone-aware delete repair, anti-entropy repair worker, query-vector cache, API batch recall, Redis-compatible shared rate limiting, explicit and batch recall feedback, Memory OS adaptive prewarm/transition-learned predictive prefetch/consolidation/forgetting plus execution-plan safety, replicated runtime, cursor-based active-active namespace delta sync, sustained active-active mesh sync, FastAPI service-region active-active sync, field-only hotness delta sync, field-state CRDT convergence, replicated snapshot/restore with offsite, archive, object-store latest-metadata/download/retention/DR-drill verification, SQLite point-in-time recovery journal replay, query-audit cache prewarm, Redis-compatible shared hot-cache behavior, namespace invalidation, API cache mutation safety on remember/feedback/feedback-batch/forget, structured/multimodal/cross-modal payload retrieval, and temporal event interval/recency retrieval with provenance, persistence, and external precomputed-vector compatibility. | implemented | Mem0 / Zep / LangGraph persistent memory / GraphRAG target adapters | Keep quorum replication, control-plane split-brain rejection, distributed namespace routing, autoscaling manifests, operator status conditions, production admission wiring, scheduled repair, service-mode repair, HTTP shard transport, sustained mixed cluster load, tombstone-aware delete repair, anti-entropy background repair, query-vector cache, API batch recall, shared rate limiting, Memory OS prewarm/transition-learned predictive prefetch/consolidation/forgetting and rollout safety contracts, explicit and batch recall feedback, cursor-bounded namespace-delta sync, sustained active-active mesh convergence, service-region active-active delta endpoints, field-state CRDT merge, repair, local and Redis cache prewarm, mutation-safe shared cache behavior, temporal/cross-modal provenance, precomputed-vector compatibility, offsite/archive/object-store backup lifecycle, SQLite point-in-time recovery, restore drills, and 10M compressed load tests green. | | Local HTTP cluster smoke | Starts real localhost API-node processes and runs the service-mode sustained mixed workload through HTTP with `read_fanout=1`, then probes `/stats` on every node for health/circuit state. | implemented | WaveMind local API nodes | Keep success, failover, repair, forget, delete suppression, and cluster health at 1.00; GitHub CI uses a 2000 ms p99 ceiling for runner variance while checked-in local evidence stays below 1000 ms. | | Local HTTP active-active service-region smoke | Starts real localhost API region processes, each serving a replicated WaveMind runtime, then syncs namespace deltas through HTTP export/import endpoints. | implemented | WaveMind local replicated API regions | Keep convergence, delete suppression, and pair-sync success at 1.00, final no-op imports at 0, and p99 below 1500 ms in CI. | @@ -2960,7 +2960,7 @@ If you already use Chroma for local memory, see the practical migration guide: - The main LongMemEval evidence result is retrieval-only. The checked-in Ollama answer-generation comparison now includes WaveMind, Chroma static, and Qdrant static over 50 questions, but it is still not a full LongMemEval leaderboard-equivalent score. - Qdrant baselines in this README use embedded local mode. Qdrant itself warns that local mode is not recommended above 20000 points; use the `qdrant-service` benchmark profile before making production latency claims. - The tuned 1M Qdrant streaming result depends on safe upsert chunking, `30` seconds wait-after-build, and `100` warmup queries. The cold 1M run misses the p99 SLO, so production Qdrant claims must specify warmup/tuning behavior. -- The Qdrant streaming path has a real service smoke, a real two-service sharded smoke, a tuned 1M passing run, and checked single-service plus sharded 10M preflight contracts. It is not a completed 10M Qdrant benchmark until `benchmarks/production_streaming_load_qdrant_10m_results.json` and `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` are produced by real runs. +- The Qdrant streaming path now has real single-service and four-service sharded 10M artifacts. These prove the tested local service topology and SLO, not multi-host or multi-region deployment. - The pgvector streaming path has a real service smoke and a checked 10M preflight contract. It is not a completed 10M pgvector benchmark until `benchmarks/production_streaming_load_pgvector_10m_results.json` is produced by a real run. - The production cost model is an engineering estimate from checked-in benchmark parameters: required replicas, target QPS, replica hourly cost, vector storage, and payload storage. It is not a cloud-provider bill and must be recalibrated for real hardware. - MTEB, MIRACL, LMEB, official VectorDBBench, and RAGBench are listed as the public benchmark roadmap, not as completed results yet. diff --git a/benchmarks/AGENT_IMPACT.md b/benchmarks/AGENT_IMPACT.md index 3bd2f22..49a7821 100644 --- a/benchmarks/AGENT_IMPACT.md +++ b/benchmarks/AGENT_IMPACT.md @@ -1,6 +1,6 @@ # WaveMind Agent Impact Leaderboard -Generated: `2026-07-11T03:37:56Z`. +Generated: `2026-07-11T05:56:44Z`. Agent-impact rows come from checked-in benchmark artifacts. They show behavioral lift on the configured tasks; they do not claim general agent success outside the listed scenarios. diff --git a/benchmarks/BENCHMARK_LEADERBOARD.md b/benchmarks/BENCHMARK_LEADERBOARD.md index 253c802..2ea1968 100644 --- a/benchmarks/BENCHMARK_LEADERBOARD.md +++ b/benchmarks/BENCHMARK_LEADERBOARD.md @@ -1,7 +1,7 @@ # WaveMind Benchmark Leaderboard Generated from `benchmarks/benchmark_matrix_results.json`. -Last refresh: `2026-07-11T03:37:56Z` from `be2cebfd776b`. +Last refresh: `2026-07-11T05:56:44Z` from `c4f786e131c8`. This is a compact reader-facing view of checked-in benchmark results. It is not a universal vector-database leaderboard: each row uses the primary quality metric for that benchmark, and latency is shown separately so quality wins are not confused with speed wins. @@ -34,7 +34,7 @@ This is a compact reader-facing view of checked-in benchmark results. It is not | area | current source | claim status | next action | |---|---|---|---| -| Artifact freshness | local matrix refresh at `2026-07-11T03:37:56Z` | source `be2cebfd776b`; audit gate enforced by `validate_benchmark_artifacts.py` | Keep weekly refresh green before public claims. | +| Artifact freshness | local matrix refresh at `2026-07-11T05:56:44Z` | source `c4f786e131c8`; audit gate enforced by `validate_benchmark_artifacts.py` | Keep weekly refresh green before public claims. | | Serverless telemetry | loopback API pool; `loopback-api-capacity-estimate`; 4 measured replicas | observed SLO `True`; loopback evidence, not a managed-serverless claim | Run `.github/workflows/serverless-observed-telemetry.yml` against deployed API nodes. | | External HTTP cluster load | kubernetes-kind-non-loopback-ci; `kubernetes-pod-dns-physical-node-drill`; 4 nodes | SLO `True`; non-loopback Kubernetes pod-DNS evidence | Run `.github/workflows/external-http-cluster-load.yml` with a remote node manifest. | | External HTTP active-active loopback | local-loopback; `loopback-api-regions`; 3 regions | SLO `True`; external URL contract over local API regions | Run `.github/workflows/external-http-active-active.yml` with remote regions for production evidence. | @@ -43,7 +43,7 @@ This is a compact reader-facing view of checked-in benchmark results. It is not | 10M streaming load | local `WaveMind faiss-ivfpq-persisted streaming` profile | target recall `0.99`, p99 `60.1 ms`, SLO `scale_required` | Repeat at 50M and add service-backed Qdrant/pgvector 10M artifacts. | | 50M streaming preflight | `WaveMind faiss-ivfpq-persisted streaming` plan-only contract | `action_required`; index `1.12 GB`; app storage `119.2 GB`; blockers `missing_env:WAVEMIND_FAISS_IVFPQ_PATH, insufficient_local_disk_for_index_and_transient_batches` | Run `.github/workflows/production-streaming-load.yml` with `faiss-ivfpq-persisted` and publish `benchmarks/production_streaming_load_ivfpq_50m_results.json`. | | Qdrant streaming | real Qdrant service smoke plus measured 10M profile | smoke recall `1`, smoke p99 `17.9 ms`; 10M recall `0.975`, 10M p99 `43.3 ms`, SLO `scale_required` | Keep the measured 10M profile green and run the sharded Qdrant and pgvector 10M profiles next. | -| Qdrant sharded streaming | real two-service fanout smoke plus horizontal Qdrant preflight | smoke recall `1`, smoke p99 `16.0 ms`; 10M preflight `action_required`; 100M preflight `action_required`; planned shards `4`; blockers `missing_env:WAVEMIND_QDRANT_URLS, insufficient_local_disk_for_index_and_transient_batches` | Run `.github/workflows/production-streaming-load.yml` with `qdrant-sharded-service` and publish `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` or `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json`. | +| Qdrant sharded streaming | real fanout smoke plus measured four-service 10M profile | smoke recall `1`, smoke p99 `16.0 ms`; 10M recall `0.993`, 10M p99 `71.3 ms`, shards `4`; 100M preflight `action_required`; planned shards `4`; blockers `none (measured artifact passes)` | Keep the measured 10M sharded profile green and run the strict 100M sharded profile next. | | Qdrant 1M streaming | real Qdrant service run before and after warmup/chunking tuning | cold p99 `3014.0 ms`; tuned recall `1`, tuned p99 `26.4 ms`, SLO `pass` | Use the tuned warmup/chunking profile for the 10M Qdrant service run. | | pgvector streaming | real PostgreSQL/pgvector service smoke plus 10M preflight | smoke recall `1`, smoke p99 `7.624 ms`; 10M preflight `action_required` | Run `.github/workflows/production-streaming-load.yml` with `pgvector-service` against sized Postgres storage. | | Production readiness gate | checked-in benchmark artifacts | `pass`; 39/39 pass | Keep the gate at readiness_score 1.0 while repeating larger service-backed runs and moving external competitor evidence into the separate adapter profile. | diff --git a/benchmarks/BENCHMARK_REPORT.md b/benchmarks/BENCHMARK_REPORT.md index d9459d8..ef5de76 100644 --- a/benchmarks/BENCHMARK_REPORT.md +++ b/benchmarks/BENCHMARK_REPORT.md @@ -1,7 +1,7 @@ # WaveMind Benchmark Report This report is generated from `benchmarks/benchmark_matrix_results.json`. -Last refresh: `2026-07-11T03:37:56Z` from `be2cebfd776b`. +Last refresh: `2026-07-11T05:56:44Z` from `c4f786e131c8`. It separates completed local runs from runner-ready public benchmarks and planned external evaluations. Planned rows are not claimed wins. They are the public proof path WaveMind must complete before stronger production claims. @@ -27,12 +27,12 @@ Planned rows are not claimed wins. They are the public proof path WaveMind must | Production load profile 100k | production-scale | implemented | Qdrant service: Recall@k 1.00, avg latency 10.3, p95 latency 19.0, p99 latency 21.3, build ms 27439.3, SLO pass, required replicas 2, autoscaled QPS 1635.0, cost status valid_slo, cost / 1M queries 1.39, monthly target cost 365.0, storage 0.24
WaveMind pgvector: Recall@k 0.74, avg latency 17.8, p95 latency 23.5, build ms 455703.7, SLO fail, required replicas 3, autoscaled QPS 945.9, cost status invalid_slo, cost / 1M queries 2.08, monthly target cost 547.5, storage 0.24
WaveMind faiss-persisted: skipped - Set WAVEMIND_FAISS_PATH to use the persisted FAISS backend | Keep at least one service backend at SLO pass while adding persisted FAISS from the Linux benchmark container and raising the target QPS. | | Production load profile 1M | production-scale | implemented | Qdrant service: Recall@k 0.98, avg latency 82.6, p95 latency 126.0, p99 latency 137.9, build ms 441775.0, SLO fail, required replicas 12, autoscaled QPS 203.5, cost status invalid_slo, cost / 1M queries 8.33, monthly target cost 2190.2, storage 2.38
WaveMind faiss-persisted: Recall@k 1.00, avg latency 39.1, p95 latency 45.3, p99 latency 57.7, build ms 20788.1, SLO scale_required, required replicas 6, autoscaled QPS 429.5, cost status valid_slo, cost / 1M queries 4.17, monthly target cost 1095.2, storage 2.38 | Keep the FAISS persisted 1M profile green and tune Qdrant/pgvector so service-mode vector DB backends also pass recall and p99. | | Qdrant 1M HNSW ef sweep | production-scale | implemented | hnsw_ef=512: Recall@k 0.75, avg latency 47.2, p95 latency 68.5, p99 latency 68.5, max latency ms 68.5, SLO fail, required replicas 7, autoscaled QPS 356.2, cost status invalid_slo, cost / 1M queries 4.86, monthly target cost 1277.7, storage 2.38
hnsw_ef=768: Recall@k 0.85, avg latency 44.0, p95 latency 69.1, p99 latency 69.8, max latency ms 69.8, SLO fail, required replicas 7, autoscaled QPS 381.5, cost status invalid_slo, cost / 1M queries 4.86, monthly target cost 1277.7, storage 2.38
hnsw_ef=1024: Recall@k 0.88, avg latency 62.9, p95 latency 81.1, p99 latency 85.5, max latency ms 85.5, SLO fail, required replicas 9, autoscaled QPS 267.0, cost status invalid_slo, cost / 1M queries 6.25, monthly target cost 1642.7, storage 2.38
hnsw_ef=1536: Recall@k 0.94, avg latency 65.6, p95 latency 111.2, p99 latency 119.7, max latency ms 119.7, SLO fail, required replicas 10, autoscaled QPS 256.3, cost status invalid_slo, cost / 1M queries 6.94, monthly target cost 1825.2, storage 2.38
hnsw_ef=2048: Recall@k 0.98, avg latency 64.8, p95 latency 91.2, p99 latency 103.8, max latency ms 103.8, SLO fail, required replicas 10, autoscaled QPS 259.4, cost status invalid_slo, cost / 1M queries 6.94, monthly target cost 1825.2, storage 2.38 | Repeat with 100+ queries and collection-level HNSW build parameters; the current best recall setting still misses the p99 SLO. | -| Production streaming load runner | production-scale | implemented | 10k smoke / WaveMind numpy-streaming: Recall@k 1, target recall@k 1, target recall@1 1, avg latency 0.10, p95 latency 0.12, p99 latency 0.46, build ms 27.6, SLO pass, required replicas 1, autoscaled QPS 170611.0, cost status valid_slo, cost / 1M queries 0.69, monthly target cost 182.5, storage 0.02, memory mode stores full matrix; smoke/testing only
100k compressed / WaveMind faiss-ivfpq-persisted streaming: Recall@k 0.96, target recall@k 0.96, target recall@1 0.96, avg latency 0.47, p95 latency 0.54, p99 latency 1.10, build ms 27322.1, SLO pass, required replicas 1, autoscaled QPS 36063.6, cost status valid_slo, cost / 1M queries 0.69, monthly target cost 182.5, storage 0.24, memory mode streaming IVF-PQ; compressed codes plus query source vectors only
1M compressed / WaveMind faiss-ivfpq-persisted streaming: Recall@k 0.99, target recall@k 0.99, target recall@1 0.98, avg latency 3.18, p95 latency 4.13, p99 latency 4.99, build ms 67255.7, SLO pass, required replicas 1, autoscaled QPS 5287.9, cost status valid_slo, cost / 1M queries 0.69, monthly target cost 182.7, storage 2.38, memory mode streaming IVF-PQ; compressed codes plus query source vectors only
10M compressed / WaveMind faiss-ivfpq-persisted streaming: Recall@k 0.99, target recall@k 0.99, target recall@1 0.95, avg latency 45.8, p95 latency 57.5, p99 latency 60.1, build ms 249349.6, SLO scale_required, required replicas 7, autoscaled QPS 366.8, cost status valid_slo, cost / 1M queries 4.86, monthly target cost 1279.9, storage 23.8, memory mode streaming IVF-PQ; compressed codes plus query source vectors only
50M preflight / WaveMind faiss-ivfpq-persisted streaming: status action_required, vectors 50000000, vector dim 128, queries 2000, estimated index 1.12, transient runner 0.57, application storage 119.2, required local free 2.12, disk free 0.00, index mode persisted FAISS IVF-PQ compressed codes plus int64 ids, required env WAVEMIND_FAISS_IVFPQ_PATH, missing env WAVEMIND_FAISS_IVFPQ_PATH, blockers missing_env:WAVEMIND_FAISS_IVFPQ_PATH, insufficient_local_disk_for_index_and_transient_batches
Qdrant smoke / Qdrant service streaming: Recall@k 1, target recall@k 1, target recall@1 1, avg latency 9.73, p95 latency 17.7, p99 latency 17.9, build ms 1002.5, SLO pass, required replicas 2, autoscaled QPS 1726.3, cost status valid_slo, cost / 1M queries 1.39, monthly target cost 365.0, storage 0.00, memory mode streaming upsert; query source vectors only
1M Qdrant cold / Qdrant service streaming: Recall@k 0.99, target recall@k 0.99, target recall@1 0.99, avg latency 161.9, p95 latency 382.5, p99 latency 3014.0, build ms 361950.1, SLO fail, required replicas 24, autoscaled QPS 103.8, cost status invalid_slo, cost / 1M queries 16.7, monthly target cost 4380.2, storage 2.38, memory mode streaming upsert; query source vectors only
1M Qdrant tuned / Qdrant service streaming: Recall@k 1, target recall@k 1, target recall@1 1, avg latency 16.2, p95 latency 24.4, p99 latency 26.4, build ms 373952.8, SLO pass, required replicas 3, autoscaled QPS 1039.3, cost status valid_slo, cost / 1M queries 2.08, monthly target cost 547.7, storage 2.38, memory mode streaming upsert; query source vectors only
10M Qdrant measured / Qdrant service streaming: Recall@k 0.97, target recall@k 0.97, target recall@1 0.97, avg latency 30.3, p95 latency 37.8, p99 latency 43.3, build ms 10214.9, SLO scale_required, required replicas 5, autoscaled QPS 554.5, cost status valid_slo, cost / 1M queries 3.47, monthly target cost 914.9, storage 23.8, memory mode streaming upsert; query source vectors only
Qdrant sharded smoke / Qdrant sharded service streaming: Recall@k 1, target recall@k 1, target recall@1 1, avg latency 9.10, p95 latency 15.6, p99 latency 16.0, build ms 3681.3, SLO pass, required replicas 2, autoscaled QPS 1845.5, cost status valid_slo, cost / 1M queries 1.39, monthly target cost 365.0, storage 0.01, shard count 2, fanout workers 2, memory mode horizontally sharded streaming upsert; parallel fanout query merge
10M Qdrant preflight / Qdrant service streaming: status action_required, vectors 10000000, vector dim 128, queries 2000, estimated index 0.00, transient runner 0.00, application storage 23.8, required local free 0.00, disk free 0.00, index mode remote Qdrant service storage; local runner stores only generated batches, required env WAVEMIND_QDRANT_URL, missing env WAVEMIND_QDRANT_URL, blockers missing_env:WAVEMIND_QDRANT_URL, insufficient_local_disk_for_index_and_transient_batches
10M Qdrant sharded preflight / Qdrant sharded service streaming: status action_required, vectors 10000000, vector dim 128, queries 2000, estimated index 0.00, transient runner 0.00, application storage 23.8, required local free 0.00, disk free 0.00, index mode remote horizontally sharded Qdrant storage; local runner routes ids across service URLs and fanout-merges top-k, required env WAVEMIND_QDRANT_URLS, missing env WAVEMIND_QDRANT_URLS, blockers missing_env:WAVEMIND_QDRANT_URLS, insufficient_local_disk_for_index_and_transient_batches
100M Qdrant sharded preflight / Qdrant sharded service streaming: status action_required, vectors 100000000, vector dim 128, queries 5000, estimated index 0.00, transient runner 0.01, application storage 238.4, required local free 0.01, disk free 0.00, index mode remote horizontally sharded Qdrant storage; local runner routes ids across service URLs and fanout-merges top-k, required env WAVEMIND_QDRANT_URLS, missing env WAVEMIND_QDRANT_URLS, blockers missing_env:WAVEMIND_QDRANT_URLS, insufficient_local_disk_for_index_and_transient_batches
pgvector smoke / WaveMind pgvector streaming: Recall@k 1, target recall@k 1, target recall@1 1, avg latency 2.59, p95 latency 4.45, p99 latency 7.62, build ms 763.6, SLO pass, required replicas 1, autoscaled QPS 6476.7, cost status valid_slo, cost / 1M queries 0.69, monthly target cost 182.5, storage 0.00, memory mode streaming PostgreSQL insert; query source vectors only
10M pgvector preflight / WaveMind pgvector streaming: status action_required, vectors 10000000, vector dim 128, queries 2000, estimated index 0.00, transient runner 0.00, application storage 23.8, required local free 0.00, disk free 0.00, index mode remote PostgreSQL/pgvector storage; local runner stores only generated batches, required env WAVEMIND_PGVECTOR_DSN, missing env WAVEMIND_PGVECTOR_DSN, blockers missing_env:WAVEMIND_PGVECTOR_DSN, insufficient_local_disk_for_index_and_transient_batches | Run .github/workflows/production-streaming-load.yml with service credentials on a sized runner for 10M Qdrant, 10M sharded Qdrant, 10M pgvector, 50M compressed FAISS, and 100M sharded Qdrant profiles. | +| Production streaming load runner | production-scale | implemented | 10k smoke / WaveMind numpy-streaming: Recall@k 1, target recall@k 1, target recall@1 1, avg latency 0.10, p95 latency 0.12, p99 latency 0.46, build ms 27.6, SLO pass, required replicas 1, autoscaled QPS 170611.0, cost status valid_slo, cost / 1M queries 0.69, monthly target cost 182.5, storage 0.02, memory mode stores full matrix; smoke/testing only
100k compressed / WaveMind faiss-ivfpq-persisted streaming: Recall@k 0.96, target recall@k 0.96, target recall@1 0.96, avg latency 0.47, p95 latency 0.54, p99 latency 1.10, build ms 27322.1, SLO pass, required replicas 1, autoscaled QPS 36063.6, cost status valid_slo, cost / 1M queries 0.69, monthly target cost 182.5, storage 0.24, memory mode streaming IVF-PQ; compressed codes plus query source vectors only
1M compressed / WaveMind faiss-ivfpq-persisted streaming: Recall@k 0.99, target recall@k 0.99, target recall@1 0.98, avg latency 3.18, p95 latency 4.13, p99 latency 4.99, build ms 67255.7, SLO pass, required replicas 1, autoscaled QPS 5287.9, cost status valid_slo, cost / 1M queries 0.69, monthly target cost 182.7, storage 2.38, memory mode streaming IVF-PQ; compressed codes plus query source vectors only
10M compressed / WaveMind faiss-ivfpq-persisted streaming: Recall@k 0.99, target recall@k 0.99, target recall@1 0.95, avg latency 45.8, p95 latency 57.5, p99 latency 60.1, build ms 249349.6, SLO scale_required, required replicas 7, autoscaled QPS 366.8, cost status valid_slo, cost / 1M queries 4.86, monthly target cost 1279.9, storage 23.8, memory mode streaming IVF-PQ; compressed codes plus query source vectors only
50M preflight / WaveMind faiss-ivfpq-persisted streaming: status action_required, vectors 50000000, vector dim 128, queries 2000, estimated index 1.12, transient runner 0.57, application storage 119.2, required local free 2.12, disk free 0.00, index mode persisted FAISS IVF-PQ compressed codes plus int64 ids, required env WAVEMIND_FAISS_IVFPQ_PATH, missing env WAVEMIND_FAISS_IVFPQ_PATH, blockers missing_env:WAVEMIND_FAISS_IVFPQ_PATH, insufficient_local_disk_for_index_and_transient_batches
Qdrant smoke / Qdrant service streaming: Recall@k 1, target recall@k 1, target recall@1 1, avg latency 9.73, p95 latency 17.7, p99 latency 17.9, build ms 1002.5, SLO pass, required replicas 2, autoscaled QPS 1726.3, cost status valid_slo, cost / 1M queries 1.39, monthly target cost 365.0, storage 0.00, memory mode streaming upsert; query source vectors only
1M Qdrant cold / Qdrant service streaming: Recall@k 0.99, target recall@k 0.99, target recall@1 0.99, avg latency 161.9, p95 latency 382.5, p99 latency 3014.0, build ms 361950.1, SLO fail, required replicas 24, autoscaled QPS 103.8, cost status invalid_slo, cost / 1M queries 16.7, monthly target cost 4380.2, storage 2.38, memory mode streaming upsert; query source vectors only
1M Qdrant tuned / Qdrant service streaming: Recall@k 1, target recall@k 1, target recall@1 1, avg latency 16.2, p95 latency 24.4, p99 latency 26.4, build ms 373952.8, SLO pass, required replicas 3, autoscaled QPS 1039.3, cost status valid_slo, cost / 1M queries 2.08, monthly target cost 547.7, storage 2.38, memory mode streaming upsert; query source vectors only
10M Qdrant measured / Qdrant service streaming: Recall@k 0.97, target recall@k 0.97, target recall@1 0.97, avg latency 30.3, p95 latency 37.8, p99 latency 43.3, build ms 10214.9, SLO scale_required, required replicas 5, autoscaled QPS 554.5, cost status valid_slo, cost / 1M queries 3.47, monthly target cost 914.9, storage 23.8, memory mode streaming upsert; query source vectors only
Qdrant sharded smoke / Qdrant sharded service streaming: Recall@k 1, target recall@k 1, target recall@1 1, avg latency 9.10, p95 latency 15.6, p99 latency 16.0, build ms 3681.3, SLO pass, required replicas 2, autoscaled QPS 1845.5, cost status valid_slo, cost / 1M queries 1.39, monthly target cost 365.0, storage 0.01, shard count 2, fanout workers 2, memory mode horizontally sharded streaming upsert; parallel fanout query merge
10M Qdrant sharded measured / Qdrant sharded service streaming: Recall@k 0.99, target recall@k 0.99, target recall@1 0.99, avg latency 46.6, p95 latency 60.5, p99 latency 71.3, build ms 5520602.2, SLO scale_required, required replicas 17, autoscaled QPS 720.9, cost status valid_slo, cost / 1M queries 4.72, monthly target cost 3104.9, storage 23.8, shard count 4, fanout workers 4, memory mode horizontally sharded streaming upsert; parallel fanout query merge
10M Qdrant preflight / Qdrant service streaming: status action_required, vectors 10000000, vector dim 128, queries 2000, estimated index 0.00, transient runner 0.00, application storage 23.8, required local free 0.00, disk free 0.00, index mode remote Qdrant service storage; local runner stores only generated batches, required env WAVEMIND_QDRANT_URL, missing env WAVEMIND_QDRANT_URL, blockers missing_env:WAVEMIND_QDRANT_URL, insufficient_local_disk_for_index_and_transient_batches
10M Qdrant sharded preflight / Qdrant sharded service streaming: status action_required, vectors 10000000, vector dim 128, queries 2000, estimated index 0.00, transient runner 0.00, application storage 23.8, required local free 0.00, disk free 10.8, index mode remote horizontally sharded Qdrant storage; local runner routes ids across service URLs and fanout-merges top-k, required env WAVEMIND_QDRANT_URLS, missing env WAVEMIND_QDRANT_URLS, blockers missing_env:WAVEMIND_QDRANT_URLS
100M Qdrant sharded preflight / Qdrant sharded service streaming: status action_required, vectors 100000000, vector dim 128, queries 5000, estimated index 0.00, transient runner 0.01, application storage 238.4, required local free 0.01, disk free 10.8, index mode remote horizontally sharded Qdrant storage; local runner routes ids across service URLs and fanout-merges top-k, required env WAVEMIND_QDRANT_URLS, missing env WAVEMIND_QDRANT_URLS, blockers missing_env:WAVEMIND_QDRANT_URLS
pgvector smoke / WaveMind pgvector streaming: Recall@k 1, target recall@k 1, target recall@1 1, avg latency 2.59, p95 latency 4.45, p99 latency 7.62, build ms 763.6, SLO pass, required replicas 1, autoscaled QPS 6476.7, cost status valid_slo, cost / 1M queries 0.69, monthly target cost 182.5, storage 0.00, memory mode streaming PostgreSQL insert; query source vectors only
10M pgvector preflight / WaveMind pgvector streaming: status action_required, vectors 10000000, vector dim 128, queries 2000, estimated index 0.00, transient runner 0.00, application storage 23.8, required local free 0.00, disk free 0.00, index mode remote PostgreSQL/pgvector storage; local runner stores only generated batches, required env WAVEMIND_PGVECTOR_DSN, missing env WAVEMIND_PGVECTOR_DSN, blockers missing_env:WAVEMIND_PGVECTOR_DSN, insufficient_local_disk_for_index_and_transient_batches | Run .github/workflows/production-streaming-load.yml with sized pgvector infrastructure for 10M, then execute the 100M sharded Qdrant profile. | | Scale readiness profile | production-scale | implemented | WaveMind cluster planner: simulated memories 1000000, namespaces 4096, nodes 4, replication factor 2, node loss min availability 1.00, zone loss min availability 1.00, read quorum 1, write quorum 2, kubernetes manifest kind StatefulSet, kubernetes repair cronjob kind CronJob, kubernetes repair cronjob namespaces 4096, placement ms 85.9
WaveMind cluster autoscaler: status scale_required, namespace count 4096, current nodes 4, required nodes 50, additional nodes 46, replication factor 3, target memories 10000000, max memories per node 1000000, headroom 0.70, current max node memories 10000000, target max node memories 678711, target within headroom True, move sample 4094, omitted moves 0, has scale action True, rebalance status ready, rebalance full plan True, rebalance batches 82, rebalance move count 4094, rebalance write quorum 2, rebalance read quorum 1, rebalance estimated steps 328, rebalance max batch node pressure 50, rebalance all batches checkpointed True, rebalance all batches repaired True, rebalance all batches validated True, plan ms 7267.0
WaveMind hot cache: queries 2000, capacity 512, hit rate 0.92, evictions 0, prewarm warmed 1, prewarm hit True, p99 lookup ms 0.01
WaveMind query vector cache: queries 200, local encode calls 1, local hit rate 0.99, Redis shared True, Redis encode calls 1, Redis reader hits 1, p99 local query ms 4.44, service boundary FastAPI TestClient, service queries 200, service encode calls 1, service saved encode calls 199, service hit rate 0.99, service metrics exposed True, service p99 latency 11.9
WaveMind API batch query: queries 100, individual HTTP requests 100, batch HTTP requests 1, request reduction 0.99, individual success True, batch success True, individual encode calls 1, batch encode calls 1, batch hit rate 0.99, batch metrics exposed True, batch total speedup 6.30, batch request latency 97.0
WaveMind shared rate limiter: backend redis-compatible fixed window, workers 2, limit per minute 4, allowed 4, limited 1, shared across workers True, expire seconds 120, p99 check ms 0.07
WaveMind Memory OS: ok True, hot queries 2, prewarm warmed 2, prewarm hit True, predictive prefetch generated 6, predictive prefetch warmed 6, transition-prefetch queries risk limits, transition-prefetch edges {'namespace': 'tenant:os', 'from_query': 'budget recall', 'to_query': 'risk limits', 'count': 1, 'probability': 1.0, 'last_seen': 1783637014.8730922}, transition-prefetch hit True, expired purged 1, concepts created 1, concept recall True, user feedback events 8, positive feedback priority delta 0.40, negative feedback priority delta -0.30, priority predictions 2, forgetting demotions 4, architecture advice architecture_required, architecture ids scale-plan, service-index, namespace-sharding, production-controls, replication-capacity, load-test, multimodal-payloads, architecture commands 10, run ms 77.4
WaveMind Redis hot cache: client redis-compatible, shared cache visible across clients True, cache prewarm warmed 1, cache prewarm cross worker hit True, memory os ok True, memory os hot queries 2, memory os prewarm warmed 2, memory os predictive generated 6, memory os predictive warmed 6, Memory OS transition-prefetch queries risk limits, Memory OS transition-prefetch edges {'namespace': 'tenant:redis-os', 'from_query': 'budget recall', 'to_query': 'risk limits', 'count': 1, 'probability': 1.0, 'last_seen': 1783637014.1264112}, Memory OS transition-prefetch hit True, memory os concepts created 1, memory os user feedback events 8, memory os positive feedback priority delta 0.40, memory os negative feedback priority delta -0.30, memory os priority predictions 2, memory os forgetting demotions 4, Memory OS architecture advice architecture_required, Memory OS architecture ids scale-plan, service-index, namespace-sharding, production-controls, replication-capacity, load-test, multimodal-payloads, memory os cross worker hit True, namespace invalidation removed True, redis keys 8, avg lookup ms 0.09, p99 lookup ms 0.10
WaveMind sustained HTTP cluster load: nodes 4, namespaces 4, replication factor 3, writes 8, queries 8, failover queries 8, write batch http requests 4, write batch individual http requests 24, write batch request reduction ratio 0.83, forget batch http requests 4, forget batch individual http requests 12, tombstone batch http requests 4, tombstone batch individual http requests 12, forget tombstone batch http requests 8, forget tombstone batch individual http requests 24, forget tombstone batch request reduction ratio 0.67, query batch http requests 3, query batch individual http requests 8, query batch request reduction ratio 0.62, failover batch http requests 2, failover batch individual http requests 8, failover batch request reduction ratio 0.75, write success rate 1.00, query hit rate 1.00, failover hit rate 1.00, delete suppression rate 1.00, repair repaired 1, success rate 1.00, p99 operation ms 891.7
WaveMind API cache mutation safety: client fastapi+redis-compatible-cache, first query cached True, cache invalidated on remember True, stale prevented after remember True, cache invalidated on feedback True, feedback demoted rejected memory True, cache invalidated on forget True, stale prevented after forget True, old recall after forget True, avg api ms 7.16, p99 api ms 8.62
WaveMind batch feedback: client fastapi+redis-compatible-cache, items 3, accepted 2, rejected 1, ok True, cache was warmed True, cache invalidated True, audit events 2, positive feedback priority delta 0.55, negative feedback priority delta -0.25, avg api ms 1.84, p99 api ms 1.84
WaveMind Kubernetes operator: bundle has crd True, bundle has operator deployment True, has service True, has statefulset True, has hpa True, has repair cronjob True, has memory os cronjob True, autoscaling min replicas 34, autoscaling max replicas 34, operator status ready True, operator status phase Ready, operator ready replicas 34, operator required replicas 34, operator capacity within headroom True, status memory os ready True, status memory os redis required True, status memory os redis configured True, operator true conditions AutoscalingReady, CapacityPlanned, ControlPlaneReady, MemoryOSReady, ProductionAdmissionReady, RebalancePlanned, RepairScheduled, ResourcesReady, autoscaling metrics cpu, memory, repair namespaces 4096, memory os calls plan True, memory os calls run True, memory os applies plan lock True, memory os blocks missing redis True
WaveMind serverless plan: has knative service True, has keda scaled object True, scale to zero True, max scale 256, target concurrency 80, uses postgres True, uses external qdrant True, uses shared cache True, safe for pod eviction True, keda scale target kind Deployment, valid keda scale target True, env has postgres dsn True, env has qdrant url True, env has redis url True
WaveMind serverless operational profile: slo pass True, requests per second 3200.0, avg request ms 80.0, p99 request ms 320.0, target p99 ms 500.0, cold start ms 900.0, cold start total ms 1220.0, cold start budget ms 3500.0, cold start budget ok True, required replicas 4, warm replicas 4, max scale 256, target concurrency 80, burst capacity rps 256000.0, scale out possible True, scale to zero safe True, external state ok True, uses postgres True, uses external qdrant True, uses shared cache True, has auth secret True, safe for pod eviction True, monthly compute cost usd 81.8, monthly budget usd 750.0, cost ok True, observed telemetry present True, observed telemetry source loopback-api-capacity-estimate, observed requests per second 37328.6, observed measured pool requests per second 583.3, observed per replica requests per second 145.8, observed measured replicas 4, observed p99 request ms 17.0, observed cold start total ms 2075.9, observed error rate 0.00, observed max replicas 22, observed scale out seconds 18.0, observed monthly compute cost usd -, observed slo pass True
WaveMind distributed sharding: nodes 3, replication factor 2, write quorum 2, read quorum 1, writes 2, recalled after primary loss True, repair repaired 1, repair ok True, recall after repair True, forget replicated deletes 2, tombstone RF 3, tombstone suppress before repair True, tombstone deleted 1, tombstone suppress after repair True, anti-entropy worker ok True, anti-entropy repaired 1, anti-entropy tombstone deleted 1, query after primary loss ms 1.49
WaveMind distributed HTTP sharding: nodes 3, replication factor 3, write quorum 2, read quorum 1, proxy bypass default True, writes 2, recalled after primary loss True, repair repaired 1, repair ok True, recall after repair True, tombstone missed delete replica records 1, tombstone suppress before repair True, tombstone deleted 1, tombstone stale records after repair 0, tombstone suppress after repair True, concurrent writes 12, concurrent write ok True, concurrent query hit rate 1.00, query after primary loss ms 16.2, concurrent ms 1311.0, repair ms 53.8
WaveMind replicated runtime: nodes 3, replication factor 3, write quorum 2, read quorum 1, recalled after node loss True, repair copied records 1, tombstone deleted 1, concurrent writes 12, concurrent write ok True, concurrent query hit rate 1.00, concurrent ms 365.7, p99 query after loss ms 3.50
WaveMind active-active delta sync: regions 2, replication factor per region 3, records imported 6, converged after bidirectional sync True, suppressed stale import after delete True, tombstone converged True, sync ms 42.5
WaveMind sustained active-active sync: regions 3, namespaces 3, replication factor per region 3, writes 18, sync cycles 5, pair syncs 90, cursor count 18, records imported 108, tombstones imported 6, deleted records 6, field keys exported 348, final noop records imported 0, final noop failed pairs 0, convergence rate 1.00, delete suppression rate 1.00, success rate 1.00, failed pairs 0, has more pairs 0, p99 sync ms 403.0, avg sync ms 216.5
WaveMind HTTP active-active service-region sync: service boundary FastAPI TestClient, api export endpoint /namespace-delta/export, api import endpoint /namespace-delta/import, regions 3, namespaces 2, replication factor per region 3, writes 6, sync cycles 4, pair syncs 48, cursor count 12, export calls 48, import calls 48, records imported 36, tombstones imported 6, deleted records 6, field keys exported 122, final noop records imported 0, final noop failed pairs 0, convergence rate 1.00, delete suppression rate 1.00, success rate 1.00, failed pairs 0, has more pairs 0, p99 sync ms 465.9, avg sync ms 342.7
WaveMind field-state CRDT: regions 3, commutative convergence True, idempotent remerge True, tombstone wins True, top key converged True, watermark convergence True, watermark actors 3, watermark health ok True, watermark health status pass, watermark missing detected True, watermark lag detected True, budget activation 5.00, merge ms 0.18
WaveMind replicated snapshot: nodes 3, manifest healthy True, offsite verified True, archive verified True, object store verified True, object store latest verified True, object store pruned 2, object store download verified True, object store drill ok True, restored files 3, recalled after restore node loss True, snapshot ms 269.9, restore ms 162.2
WaveMind structured payloads: queries 7, precision@1 1.00, cross-modal queries 7, cross-modal precision@1 1.00, cross-modal dim 64, cross-modal vectors persisted 1.00, cross-modal provenance 1.00, precomputed-vector queries 4, precomputed-vector precision@1 1.00, precomputed-vector dim 4, precomputed-vector persisted 1.00, encoder contract True, encoder contract payloads 7, encoder target precision@1 1.00, encoder global precision@1 1.00, encoder routing 1.00, encoder vectors persisted 1.00, encoder vectors normalized 1.00, encoder vectors finite 1.00, encoder provenance 1.00, encoder min margin 0.81, encoder health ok True, encoder health encoder descriptor, encoder health payloads 7, encoder health queries 7, encoder health global precision at 1 1.00, encoder health target modality routing rate 1.00, encoder health finite payload vector rate 1.00, encoder health normalized payload vector rate 1.00, encoder health finite query vector rate 1.00, encoder health normalized query vector rate 1.00, encoder health dimension match rate 1.00, encoder health payload encode p95 ms 8.33, encoder health query encode p95 ms 1.52, encoder health min global margin 0.25, encoder health min required margin 0.01, temporal-event queries 4, temporal-event precision@1 1.00, temporal around@1 1, temporal window@1 1, temporal recency@1 1, temporal interval@1 1, temporal persistence 1.00, temporal provenance 1.00, knowledge-graph queries 4, knowledge-graph precision@1 1.00, knowledge-graph path precision@1 1.00, knowledge-graph direct@1 1, knowledge-graph two-hop@1 1, knowledge-graph three-hop@1 1, knowledge-graph predicate@1 1, knowledge-graph persistence 1.00, knowledge-graph provenance 1.00, avg latency 1.96, p99 latency 4.91, cross-modal avg latency 4.71, cross-modal p99 latency 6.71, precomputed-vector avg latency 1.87, precomputed-vector p99 latency 2.08, temporal avg latency 2.57, temporal p99 latency 5.28, knowledge-graph avg latency 2.34, knowledge-graph p99 latency 4.36
WaveMind 100M capacity envelope: target memories 100000000, namespace count 32768, node count 128, zones 8, replication factor 3, write quorum 2, node loss min availability 1.00, zone loss min availability 1.00, replica load skew 1.09, primary load skew 1.18, max storage per node gb 5.81, recommended autoscaling max replicas 192, valid capacity plan True, placement ms 53699.9 | Move from deterministic 100M capacity planning to service-backed 100M Qdrant/pgvector/FAISS load tests on sized hardware. | | Postgres PITR runbook/preflight | production-ops | implemented | WaveMind Postgres PITR preflight: status ready, environment status missing_env, command count 7, retention hours 72, missing env WAVEMIND_POSTGRES_DSN, WAVEMIND_POSTGRES_BASEBACKUP_DIR, WAVEMIND_POSTGRES_WAL_ARCHIVE_DIR, WAVEMIND_POSTGRES_RESTORE_DATA_DIR, WAVEMIND_POSTGRES_RESTORE_TARGET_TIME, ok True | Execute the same runbook against staging or managed Postgres and commit a real PITR drill report with replay LSN, target timestamp, restore duration, and post-restore row/index checks. | | Production readiness gate | production-scale | implemented | WaveMind production readiness: readiness score 1.00, overall status pass, passed criteria 39, action required 0, failed criteria 0, total criteria 39 | Keep the gate at readiness_score 1.0 while repeating larger service-backed runs and moving external competitor evidence into the separate adapter profile. | | Production evidence environment contract | production-ops | implemented | WaveMind production evidence env: overall status action_required, required env count 9, configured required count 0, missing required count 9, recommended missing count 3, workflow count 4, dispatch job count 8, scale gap profile count 5 | Set the missing environment-scoped GitHub secrets, run the safe dispatch commands, and ingest downloaded artifacts through the strict evidence gate. | -| Strict evidence readiness runbook | production-scale | implemented | WaveMind strict evidence readiness: status pass, readiness status action_required, claim status claims_limited, total requirements 8, action required 5, ready for safe dispatch count 0, can auto run now count 0, target memories total 180000000 | Provision the missing remote/service environments, run safe dispatch commands with commit_results=false, ingest downloaded artifacts, then rerun strict evidence validation before changing release claims. | +| Strict evidence readiness runbook | production-scale | implemented | WaveMind strict evidence readiness: status pass, readiness status action_required, claim status claims_limited, total requirements 8, action required 4, ready for safe dispatch count 0, can auto run now count 0, target memories total 180000000 | Provision the missing remote/service environments, run safe dispatch commands with commit_results=false, ingest downloaded artifacts, then rerun strict evidence validation before changing release claims. | | Local HTTP cluster smoke | production-scale | implemented | WaveMind local HTTP cluster smoke: nodes 4, namespaces 4, memories per namespace 2, replication factor 3, read fanout 1, workers 4, success rate 1.00, write success rate 1.00, query hit rate 1.00, failover hit rate 1.00, delete suppression rate 1.00, repair repaired 1, p99 operation ms 348.8, slo pass True | Run the same workload against external service nodes and then increase namespace count and payload size on sized hardware. | | Local HTTP active-active service-region smoke | production-scale | implemented | WaveMind real HTTP active-active service-region sync: region count 3, namespaces 2, writes 6, sync cycles 3, pair syncs 36, cursor count 12, records imported 36, tombstones imported 6, deleted records 6, field keys exported 100, final noop records imported 0, final noop failed pairs 0, convergence rate 1.00, delete suppression rate 1.00, success rate 1.00, failed pairs 0, has more pairs 0, avg sync ms 525.4, p99 sync ms 925.2, avg operation ms 53.5, p99 operation ms 347.6, slo pass True | Run the same active-active service-region workload against remote Kubernetes/serverless API nodes with external Postgres/Qdrant/Redis state. | | External HTTP cluster load runner | production-scale | implemented | WaveMind external HTTP cluster load: nodes 4, namespaces 32, memories per namespace 8, replication factor 3, read fanout 1, workers 8, success rate 1.00, write success rate 1.00, query hit rate 1.00, failover hit rate 1.00, delete suppression rate 1.00, repair repaired 1, p99 operation ms 6694.8, slo pass True, batch query success True, batch query size 24, batch query http requests 24 -> 1, batch query request reduction ratio 0.96, batch query p99 ms 148.8, batch query total speedup 1.26 | Replace the current loopback service-node artifact with a remote node manifest run from a multi-node deployment. | diff --git a/benchmarks/COST_EFFICIENCY.md b/benchmarks/COST_EFFICIENCY.md index 5b1ab62..59887fe 100644 --- a/benchmarks/COST_EFFICIENCY.md +++ b/benchmarks/COST_EFFICIENCY.md @@ -1,16 +1,16 @@ # WaveMind Cost Efficiency Leaderboard -Generated: `2026-07-11T03:37:56Z`. +Generated: `2026-07-11T05:56:44Z`. Measured rows come from checked-in load artifacts. Planned rows are capacity and cost contracts only; they do not unlock production latency or recall claims until the matching benchmark result exists. ## Summary -- Measured rows: `21`. +- Measured rows: `22`. - Measured SLO pass rows: `9`. -- Measured valid cost rows: `12`. +- Measured valid cost rows: `13`. - Planned cost rows: `5`. -- Measured frontier: `sub_100k-wavemind-numpy-streaming-production_streaming_load_smoke_results, 100k-qdrant-service-production_load_qdrant_100k_tuned_results, 1m-qdrant-service-streaming-production_streaming_load_qdrant_1m_tuned_results, 1m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_1m_results, 100k-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_100k_results, 10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results, 10m-qdrant-service-streaming-production_streaming_load_qdrant_10m_results`. +- Measured frontier: `sub_100k-wavemind-numpy-streaming-production_streaming_load_smoke_results, 100k-qdrant-service-production_load_qdrant_100k_tuned_results, 1m-qdrant-service-streaming-production_streaming_load_qdrant_1m_tuned_results, 1m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_1m_results, 100k-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_100k_results, 10m-qdrant-sharded-service-streaming-production_streaming_load_qdrant_sharded_10m_results, 10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results, 10m-qdrant-service-streaming-production_streaming_load_qdrant_10m_results`. - Planned frontier: `qdrant-sharded-100m, faiss-ivfpq-50m`. ## Measured Cost Frontier @@ -27,8 +27,8 @@ Measured rows come from checked-in load artifacts. Planned rows are capacity and | 8 | 1m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_1m_results | 1m | WaveMind faiss-ivfpq-persisted streaming | 1,000,000 | 0.99 | 4.992 | pass | $0.694 | $182.74 | `benchmarks/production_streaming_load_ivfpq_1m_results.json` | | 9 | 100k-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_100k_results | 100k | WaveMind faiss-ivfpq-persisted streaming | 100,000 | 0.96 | 1.104 | pass | $0.694 | $182.52 | `benchmarks/production_streaming_load_ivfpq_100k_results.json` | | 10 | 1m-wavemind-faiss-persisted-production_load_faiss_1m_results | 1m | WaveMind faiss-persisted | 1,000,000 | 1 | 57.71 | scale_required | $4.167 | $1,095.24 | `benchmarks/production_load_faiss_1m_results.json` | -| 11 | 10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results | 10m | WaveMind faiss-ivfpq-persisted streaming | 10,000,000 | 0.99 | 60.13 | scale_required | $4.861 | $1,279.88 | `benchmarks/production_streaming_load_ivfpq_10m_results.json` | -| 12 | 10m-qdrant-service-streaming-production_streaming_load_qdrant_10m_results | 10m | Qdrant service streaming | 10,000,000 | 0.975 | 43.27 | scale_required | $3.472 | $914.88 | `benchmarks/production_streaming_load_qdrant_10m_results.json` | +| 11 | 10m-qdrant-sharded-service-streaming-production_streaming_load_qdrant_sharded_10m_results | 10m | Qdrant sharded service streaming | 10,000,000 | 0.993 | 71.28 | scale_required | $4.722 | $3,104.88 | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | +| 12 | 10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results | 10m | WaveMind faiss-ivfpq-persisted streaming | 10,000,000 | 0.99 | 60.13 | scale_required | $4.861 | $1,279.88 | `benchmarks/production_streaming_load_ivfpq_10m_results.json` | ## Planned Cost Frontier diff --git a/benchmarks/PRODUCTION_EVIDENCE.md b/benchmarks/PRODUCTION_EVIDENCE.md index b99f4d8..7143124 100644 --- a/benchmarks/PRODUCTION_EVIDENCE.md +++ b/benchmarks/PRODUCTION_EVIDENCE.md @@ -7,8 +7,8 @@ multi-region, managed-serverless, 50M, or 100M production scale cannot. | metric | value | |---|---:| | overall status | `action_required` | -| passed requirements | `3` | -| action required | `5` | +| passed requirements | `4` | +| action required | `4` | | failed requirements | `0` | | total requirements | `8` | @@ -18,7 +18,7 @@ multi-region, managed-serverless, 50M, or 100M production scale cannot. | External HTTP active-active regions | `action_required` | no checked-in external HTTP active-active region result; issues: missing artifact | `benchmarks/external_http_active_active_results.json` | `gh workflow run external-http-active-active.yml -f regions="us-east=https://wm-us.example.com,eu-west=https://wm-eu.example.com,ap-south=https://wm-ap.example.com" -f namespace_count=16 -f p99_slo_ms=1500 -f fail_on_slo=true -f commit_results=true` | Remote multi-region active-active memory convergence. | | Managed/serverless remote telemetry | `action_required` | missing remote serverless telemetry; issues: missing artifact | `deploy/serverless/observed-telemetry.remote.json` | `gh workflow run serverless-observed-telemetry.yml -f nodes="https://wm-a.example.com,https://wm-b.example.com" -f seed_mode=first -f commit_results=true` | Hosted/serverless p99, cold-start, error-rate, and scale-out SLO. | | 10M Qdrant service load | `pass` | Qdrant service streaming: vectors 10000000, recall 0.975, p99 43.266599997878075 ms, cost valid_slo, source be2cebfd776b, run local-qdrant10m-quantized-ef1024-v1182-20260711 | `benchmarks/production_streaming_load_qdrant_10m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 100.0 --replicas 3 --autoscaling-max-replicas 24 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_10m_results.json --checkpoint-path state/production-runs/qdrant-service-10000000.checkpoint.json` | 10M Qdrant service-backed candidate index SLO. | -| 10M sharded Qdrant service load | `action_required` | no checked-in 10,000,000-vector result for Qdrant sharded service streaming; issues: missing artifact | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json` | Horizontally sharded Qdrant service recall/latency SLO. | +| 10M sharded Qdrant service load | `pass` | Qdrant sharded service streaming: vectors 10000000, recall 0.9925, p99 71.27849999233149 ms, cost valid_slo, source 11d0be8c62e8, run local-qdrant-sharded10m-4x-v1182-20260711 | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json` | Horizontally sharded Qdrant service recall/latency SLO. | | 10M pgvector service load | `action_required` | no checked-in 10,000,000-vector result for WaveMind pgvector streaming; issues: missing artifact | `benchmarks/production_streaming_load_pgvector_10m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines pgvector-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 100.0 --replicas 3 --autoscaling-max-replicas 24 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_pgvector_10m_results.json --checkpoint-path state/production-runs/pgvector-service-10000000.checkpoint.json` | 10M PostgreSQL/pgvector service candidate-index SLO. | | 50M FAISS IVF-PQ streaming load | `pass` | WaveMind faiss-ivfpq-persisted streaming: vectors 50000000, recall 0.9705, p99 73.10794899996154 ms, cost valid_slo, source a9ea9d7f8027, run github-actions-29067655616 | `benchmarks/production_streaming_load_ivfpq_50m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 50000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 1000000 --engines faiss-ivfpq-persisted --target-recall 0.95 --target-p99-ms 100.0 --target-qps 100.0 --replicas 3 --autoscaling-max-replicas 24 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_ivfpq_50m_results.json --checkpoint-path state/production-runs/faiss-ivfpq-persisted-50000000.checkpoint.json` | 50M compressed local/persistent FAISS profile. | | 100M remote load result | `action_required` | no checked-in 100,000,000-vector result for Qdrant sharded service streaming; issues: missing artifact | `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json` | `python benchmarks/production_streaming_load_benchmark.py --sizes 100000000 --dim 128 --queries 5000 --top-k 10 --seed 42 --noise 0.08 --batch-size 10000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 500.0 --replicas 8 --autoscaling-max-replicas 128 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_100m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-100000000.checkpoint.json` | 100M+ memories with measured sharded-Qdrant recall, p99, and cost SLO. | diff --git a/benchmarks/PRODUCTION_EVIDENCE_BUNDLE.md b/benchmarks/PRODUCTION_EVIDENCE_BUNDLE.md index b74862e..c463035 100644 --- a/benchmarks/PRODUCTION_EVIDENCE_BUNDLE.md +++ b/benchmarks/PRODUCTION_EVIDENCE_BUNDLE.md @@ -7,7 +7,7 @@ claim boundaries, and the exact next actions required to unlock blocked claims. | metric | value | |---|---:| | claim status | `claims_limited` | -| strict evidence | `3/8` | +| strict evidence | `4/8` | | preflight ready | `0/8` | | production readiness | `pass` | | readiness score | `1.0` | @@ -16,7 +16,7 @@ claim boundaries, and the exact next actions required to unlock blocked claims. | production scale run contract | `available` | | production scale profiles | `5` | | production scale target memories | `180000000` | -| next actions | `5` | +| next actions | `4` | ## Claim Boundaries @@ -44,6 +44,5 @@ claim boundaries, and the exact next actions required to unlock blocked claims. |---|---|---|---|---|---| | External HTTP active-active regions | `action_required` | `action_required` | `benchmarks/external_http_active_active_results.json` | `WAVEMIND_ACTIVE_ACTIVE_REGIONS, WAVEMIND_ACTIVE_ACTIVE_REGIONS_MANIFEST_JSON; issues: missing artifact` | `gh workflow run external-http-active-active.yml -f regions="$WAVEMIND_ACTIVE_ACTIVE_REGIONS" -f commit_results=true` | | Managed/serverless remote telemetry | `action_required` | `action_required` | `deploy/serverless/observed-telemetry.remote.json` | `WAVEMIND_SERVERLESS_NODES; issues: missing artifact` | `gh workflow run serverless-observed-telemetry.yml -f nodes="$WAVEMIND_SERVERLESS_NODES" -f seed_mode=first -f commit_results=true` | -| 10M sharded Qdrant service load | `action_required` | `action_required` | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS; issues: missing artifact` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json` | | 10M pgvector service load | `action_required` | `action_required` | `benchmarks/production_streaming_load_pgvector_10m_results.json` | `WAVEMIND_PGVECTOR_DSN; issues: missing artifact` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines pgvector-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 100.0 --replicas 3 --autoscaling-max-replicas 24 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_pgvector_10m_results.json --checkpoint-path state/production-runs/pgvector-service-10000000.checkpoint.json` | | 100M remote load result | `action_required` | `action_required` | `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json` | `WAVEMIND_QDRANT_URLS; issues: missing artifact` | `python benchmarks/production_streaming_load_benchmark.py --sizes 100000000 --dim 128 --queries 5000 --top-k 10 --seed 42 --noise 0.08 --batch-size 10000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 500.0 --replicas 8 --autoscaling-max-replicas 128 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_100m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-100000000.checkpoint.json` | diff --git a/benchmarks/PRODUCTION_EVIDENCE_DISPATCH.md b/benchmarks/PRODUCTION_EVIDENCE_DISPATCH.md index 762cb32..e0d95b5 100644 --- a/benchmarks/PRODUCTION_EVIDENCE_DISPATCH.md +++ b/benchmarks/PRODUCTION_EVIDENCE_DISPATCH.md @@ -9,8 +9,8 @@ strict production-evidence validation. |---|---:| | overall status | `action_required` | | ready to dispatch | `0` | -| blocked by preflight | `5` | -| complete | `3` | +| blocked by preflight | `4` | +| complete | `4` | | total jobs | `8` | | runner label | `self-hosted-large` | | commit results default | `False` | @@ -23,7 +23,7 @@ strict production-evidence validation. | External HTTP active-active regions | `blocked_by_preflight` | `remote-service` | `external-http-active-active.yml` | `benchmarks/external_http_active_active_results.json` | `WAVEMIND_ACTIVE_ACTIVE_REGIONS, WAVEMIND_ACTIVE_ACTIVE_REGIONS_MANIFEST_JSON` | | Managed/serverless remote telemetry | `blocked_by_preflight` | `remote-service` | `serverless-observed-telemetry.yml` | `deploy/serverless/observed-telemetry.remote.json` | `WAVEMIND_SERVERLESS_NODES` | | 10M Qdrant service load | `complete` | `service-scale-10m` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_qdrant_10m_results.json` | `WAVEMIND_QDRANT_URL` | -| 10M sharded Qdrant service load | `blocked_by_preflight` | `service-scale-10m` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS` | +| 10M sharded Qdrant service load | `complete` | `service-scale-10m` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS` | | 10M pgvector service load | `blocked_by_preflight` | `service-scale-10m` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_pgvector_10m_results.json` | `WAVEMIND_PGVECTOR_DSN` | | 50M FAISS IVF-PQ streaming load | `complete` | `large-local-index` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_ivfpq_50m_results.json` | `WAVEMIND_FAISS_IVFPQ_PATH` | | 100M remote load result | `blocked_by_preflight` | `hundred-million-service` | `production-streaming-load.yml` | `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json` | `WAVEMIND_QDRANT_URLS` | diff --git a/benchmarks/PRODUCTION_READINESS.md b/benchmarks/PRODUCTION_READINESS.md index 64bba8f..91ffc95 100644 --- a/benchmarks/PRODUCTION_READINESS.md +++ b/benchmarks/PRODUCTION_READINESS.md @@ -14,7 +14,7 @@ verdict, not a marketing claim. | criterion | status | evidence | next step | |---|---|---|---| -| Checked-in benchmark artifacts are synchronized | `pass` | audit status pass, generated_at 2026-07-11T03:28:43Z | Keep the benchmark refresh workflow green and block stale artifacts before release. | +| Checked-in benchmark artifacts are synchronized | `pass` | audit status pass, generated_at 2026-07-11T05:53:28Z | Keep the benchmark refresh workflow green and block stale artifacts before release. | | Agent coherence benchmark proves behavioral lift | `pass` | WaveMind success 0.917, Chroma static 0.333, Static vector 0.333, stale error 0.000, context saved 0.931, coherent turn rate 0.750, avg latency 2.647 ms | Keep agent-behavior quality gated in CI and extend it with LLM answer-quality runs on LoCoMo/LongMemEval. | | LongMemEval answer generation beats static RAG baselines | `pass` | ollama qwen2.5:1.5b, queries 50, exact 0.240, contains 0.380, token F1 0.333, answered 0.520, grounded 0.520, supported 1.000, unsupported 0.000, faithful 1.000, abstain 0.480, evidence recall 0.920, retrieval 36.586 ms, Chroma F1 0.170, Qdrant F1 0.170 | Scale this from the checked 50-query local run to full LongMemEval-S with stronger local/API models and faithfulness scoring. | | 100k service-backed load profile passes SLO and cost gate | `pass` | recall 1.0, p99 21.25629998045042 ms, cost $1.39/1M queries | Keep the 100k profile green while adding persisted FAISS and pgvector service runs. | diff --git a/benchmarks/RELEASE_CLAIMS.md b/benchmarks/RELEASE_CLAIMS.md index a477f68..27ff2ea 100644 --- a/benchmarks/RELEASE_CLAIMS.md +++ b/benchmarks/RELEASE_CLAIMS.md @@ -13,7 +13,7 @@ until strict external evidence artifacts pass. | artifact audit | `pass` | | allowed claims | `3` | | locked claims | `2` | -| next actions | `5` | +| next actions | `4` | ## Allowed Claims @@ -36,6 +36,5 @@ until strict external evidence artifacts pass. |---|---|---|---|---|---| | External HTTP active-active regions | `action_required` | `action_required` | `benchmarks/external_http_active_active_results.json` | `WAVEMIND_ACTIVE_ACTIVE_REGIONS, WAVEMIND_ACTIVE_ACTIVE_REGIONS_MANIFEST_JSON` | `gh workflow run external-http-active-active.yml -f regions="$WAVEMIND_ACTIVE_ACTIVE_REGIONS" -f commit_results=true` | | Managed/serverless remote telemetry | `action_required` | `action_required` | `deploy/serverless/observed-telemetry.remote.json` | `WAVEMIND_SERVERLESS_NODES` | `gh workflow run serverless-observed-telemetry.yml -f nodes="$WAVEMIND_SERVERLESS_NODES" -f seed_mode=first -f commit_results=true` | -| 10M sharded Qdrant service load | `action_required` | `action_required` | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json` | | 10M pgvector service load | `action_required` | `action_required` | `benchmarks/production_streaming_load_pgvector_10m_results.json` | `WAVEMIND_PGVECTOR_DSN` | `python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines pgvector-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 100.0 --replicas 3 --autoscaling-max-replicas 24 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_pgvector_10m_results.json --checkpoint-path state/production-runs/pgvector-service-10000000.checkpoint.json` | | 100M remote load result | `action_required` | `action_required` | `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json` | `WAVEMIND_QDRANT_URLS` | `python benchmarks/production_streaming_load_benchmark.py --sizes 100000000 --dim 128 --queries 5000 --top-k 10 --seed 42 --noise 0.08 --batch-size 10000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 500.0 --replicas 8 --autoscaling-max-replicas 128 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_100m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-100000000.checkpoint.json` | diff --git a/benchmarks/SCALE_GAP.md b/benchmarks/SCALE_GAP.md index 5848f78..2c7e91d 100644 --- a/benchmarks/SCALE_GAP.md +++ b/benchmarks/SCALE_GAP.md @@ -7,18 +7,18 @@ are proven, which are plan-only, and what must run next. | metric | value | |---|---:| | overall status | `action_required` | -| complete profiles | `2/5` | +| complete profiles | `3/5` | | ready to run | `0` | -| blocked by env | `3` | +| blocked by env | `2` | | planned target memories | `180000000` | -| proven target memories | `60000000` | +| proven target memories | `70000000` | | nearest baseline max memories | `10000000` | | claim status | `claims_limited` | | profile | status | target | nearest baseline | gap | artifact | missing env | |---|---|---:|---:|---:|---|---| | qdrant-10m | `complete` | 10000000 | 1000000 | 10.0 | `benchmarks/production_streaming_load_qdrant_10m_results.json` | `WAVEMIND_QDRANT_URL` | -| qdrant-sharded-10m | `blocked_by_env` | 10000000 | 1000000 | 10.0 | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS` | +| qdrant-sharded-10m | `complete` | 10000000 | 1000000 | 10.0 | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS` | | pgvector-10m | `blocked_by_env` | 10000000 | 50000 | 200.0 | `benchmarks/production_streaming_load_pgvector_10m_results.json` | `WAVEMIND_PGVECTOR_DSN` | | faiss-ivfpq-50m | `complete` | 50000000 | 10000000 | 5.0 | `benchmarks/production_streaming_load_ivfpq_50m_results.json` | `WAVEMIND_FAISS_IVFPQ_PATH` | | qdrant-sharded-100m | `blocked_by_env` | 100000000 | 1000000 | 100.0 | `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json` | `WAVEMIND_QDRANT_URLS` | diff --git a/benchmarks/STRICT_EVIDENCE_READINESS.md b/benchmarks/STRICT_EVIDENCE_READINESS.md index 0fd2e4e..54f4322 100644 --- a/benchmarks/STRICT_EVIDENCE_READINESS.md +++ b/benchmarks/STRICT_EVIDENCE_READINESS.md @@ -11,7 +11,7 @@ production evidence by itself. | readiness status | `action_required` | | claim status | `claims_limited` | | total requirements | `8` | -| action required | `5` | +| action required | `4` | | ready for safe dispatch | `0` | | can auto-run now | `0` | | planned target memories | `180000000` | @@ -37,7 +37,7 @@ production evidence by itself. | External HTTP active-active regions | `missing_env` | `blocked_by_preflight` | | `benchmarks/external_http_active_active_results.json` | `WAVEMIND_ACTIVE_ACTIVE_REGIONS, WAVEMIND_ACTIVE_ACTIVE_REGIONS_MANIFEST_JSON` | Remote multi-region active-active convergence | | Managed/serverless remote telemetry | `missing_env` | `blocked_by_preflight` | | `deploy/serverless/observed-telemetry.remote.json` | `WAVEMIND_SERVERLESS_NODES` | Hosted/serverless p99, cold-start, error-rate, and scale-out SLO. | | 10M Qdrant service load | `complete` | `complete` | 10000000 | `benchmarks/production_streaming_load_qdrant_10m_results.json` | `WAVEMIND_QDRANT_URL` | 10M-100M service-backed production scale | -| 10M sharded Qdrant service load | `missing_env` | `blocked_by_preflight` | 10000000 | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS` | 10M-100M service-backed production scale | +| 10M sharded Qdrant service load | `complete` | `complete` | 10000000 | `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` | `WAVEMIND_QDRANT_URLS` | 10M-100M service-backed production scale | | 10M pgvector service load | `missing_env` | `blocked_by_preflight` | 10000000 | `benchmarks/production_streaming_load_pgvector_10m_results.json` | `WAVEMIND_PGVECTOR_DSN` | 10M-100M service-backed production scale | | 50M FAISS IVF-PQ streaming load | `complete` | `complete` | 50000000 | `benchmarks/production_streaming_load_ivfpq_50m_results.json` | `WAVEMIND_FAISS_IVFPQ_PATH` | 10M-100M service-backed production scale | | 100M remote load result | `missing_env` | `blocked_by_preflight` | 100000000 | `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json` | `WAVEMIND_QDRANT_URLS` | 10M-100M service-backed production scale | diff --git a/benchmarks/agent_impact_results.json b/benchmarks/agent_impact_results.json index e5d4be7..cedacca 100644 --- a/benchmarks/agent_impact_results.json +++ b/benchmarks/agent_impact_results.json @@ -1,7 +1,7 @@ { "schema": "wavemind.agent_impact_leaderboard.v1", - "generated_at": "2026-07-11T03:37:56Z", - "source_ref": "be2cebfd776b", + "generated_at": "2026-07-11T05:56:44Z", + "source_ref": "c4f786e131c8", "claim_boundary": "Agent-impact rows come from checked-in benchmark artifacts. They show behavioral lift on the configured tasks; they do not claim general agent success outside the listed scenarios.", "summary": { "benchmark_count": 6, diff --git a/benchmarks/benchmark_artifact_audit.json b/benchmarks/benchmark_artifact_audit.json index ab90a7f..57e91dd 100644 --- a/benchmarks/benchmark_artifact_audit.json +++ b/benchmarks/benchmark_artifact_audit.json @@ -1,13 +1,13 @@ { "schema": "wavemind.benchmark_artifact_audit.v1", "status": "pass", - "checked_at": "2026-07-11T03:38:05Z", - "generated_at": "2026-07-11T03:37:56Z", - "source_ref": "be2cebfd776b", + "checked_at": "2026-07-11T05:56:54Z", + "generated_at": "2026-07-11T05:56:44Z", + "source_ref": "c4f786e131c8", "workflow_run_id": null, "refresh_profile": "local", "max_age_days": 8.0, - "age_days": 0.00010598428240740742, + "age_days": 0.00012261520833333332, "implemented_count": 37, "runner_ready_count": 3, "planned_count": 6, diff --git a/benchmarks/benchmark_matrix_results.json b/benchmarks/benchmark_matrix_results.json index 68a69b9..d0be8ae 100644 --- a/benchmarks/benchmark_matrix_results.json +++ b/benchmarks/benchmark_matrix_results.json @@ -1,7 +1,7 @@ { "schema": "wavemind.benchmark_matrix.v1", - "generated_at": "2026-07-11T03:37:56Z", - "source_ref": "be2cebfd776b", + "generated_at": "2026-07-11T05:56:44Z", + "source_ref": "c4f786e131c8", "workflow_run_id": null, "refresh_profile": "local", "note": "Implemented entries are runnable from this repository. Planned entries are public benchmarks that require optional datasets, services, or heavier dependencies.", @@ -1171,6 +1171,25 @@ "fanout_workers": 2, "memory_mode": "horizontally sharded streaming upsert; parallel fanout query merge" }, + "10M Qdrant sharded measured / Qdrant sharded service streaming": { + "recall_at_k": 0.9925, + "target_recall_at_k": 0.9925, + "target_recall_at_1": 0.9925, + "avg_latency_ms": 46.60563879973779, + "p95_latency_ms": 60.49189998884685, + "p99_latency_ms": 71.27849999233149, + "build_ms": 5520602.228099975, + "slo_status": "scale_required", + "slo_required_replicas": 17, + "slo_autoscaled_qps": 720.9428057488408, + "cost_status": "valid_slo", + "compute_cost_per_1m_queries_usd": 4.722222222222222, + "monthly_total_cost_at_target_qps_usd": 3104.8841857910156, + "estimated_storage_gb": 23.84185791015625, + "shard_count": 4, + "fanout_workers": 4, + "memory_mode": "horizontally sharded streaming upsert; parallel fanout query merge" + }, "10M Qdrant preflight / Qdrant service streaming": { "status": "action_required", "vectors": 10000000, @@ -1202,7 +1221,7 @@ "estimated_transient_runner_gb": 0.003, "estimated_application_storage_gb": 23.842, "required_local_free_gb": 0.004, - "disk_free_gb": 0.0, + "disk_free_gb": 10.771, "index_mode": "remote horizontally sharded Qdrant storage; local runner routes ids across service URLs and fanout-merges top-k", "required_env": [ "WAVEMIND_QDRANT_URLS" @@ -1211,8 +1230,7 @@ "WAVEMIND_QDRANT_URLS" ], "blockers": [ - "missing_env:WAVEMIND_QDRANT_URLS", - "insufficient_local_disk_for_index_and_transient_batches" + "missing_env:WAVEMIND_QDRANT_URLS" ] }, "100M Qdrant sharded preflight / Qdrant sharded service streaming": { @@ -1224,7 +1242,7 @@ "estimated_transient_runner_gb": 0.007, "estimated_application_storage_gb": 238.419, "required_local_free_gb": 0.009, - "disk_free_gb": 0.0, + "disk_free_gb": 10.771, "index_mode": "remote horizontally sharded Qdrant storage; local runner routes ids across service URLs and fanout-merges top-k", "required_env": [ "WAVEMIND_QDRANT_URLS" @@ -1233,8 +1251,7 @@ "WAVEMIND_QDRANT_URLS" ], "blockers": [ - "missing_env:WAVEMIND_QDRANT_URLS", - "insufficient_local_disk_for_index_and_transient_batches" + "missing_env:WAVEMIND_QDRANT_URLS" ] }, "pgvector smoke / WaveMind pgvector streaming": { @@ -1277,8 +1294,8 @@ ] } }, - "target": "Keep 10M compressed FAISS IVF-PQ above recall@10 0.95 and p99 below 100 ms, keep tuned 1M Qdrant streaming below p99 100 ms, keep Qdrant, sharded Qdrant, and pgvector streaming smokes green, keep 10M single-service Qdrant, 10M sharded Qdrant, 10M pgvector, 50M FAISS, and 100M sharded Qdrant preflights reproducible, then run those service profiles on hardware sized for the index.", - "next_step": "Run .github/workflows/production-streaming-load.yml with service credentials on a sized runner for 10M Qdrant, 10M sharded Qdrant, 10M pgvector, 50M compressed FAISS, and 100M sharded Qdrant profiles." + "target": "Keep strict 10M single-service and four-service sharded Qdrant, 10M/50M compressed FAISS, and service smokes green while completing 10M pgvector and 100M sharded Qdrant evidence.", + "next_step": "Run .github/workflows/production-streaming-load.yml with sized pgvector infrastructure for 10M, then execute the 100M sharded Qdrant profile." }, { "id": "scale_readiness", @@ -2013,7 +2030,7 @@ "readiness_status": "action_required", "claim_status": "claims_limited", "total_requirements": 8, - "action_required_count": 5, + "action_required_count": 4, "ready_for_safe_dispatch_count": 0, "can_auto_run_now_count": 0, "target_memories_total": 180000000 diff --git a/benchmarks/benchmark_registry.py b/benchmarks/benchmark_registry.py index 59bdf31..deec34e 100644 --- a/benchmarks/benchmark_registry.py +++ b/benchmarks/benchmark_registry.py @@ -560,6 +560,7 @@ def _implemented_entries(root: Path) -> list[dict[str, Any]]: production_streaming_qdrant_1m_tuned_payload = _load_json(root / "benchmarks" / "production_streaming_load_qdrant_1m_tuned_results.json") production_streaming_qdrant_10m_payload = _load_json(root / "benchmarks" / "production_streaming_load_qdrant_10m_results.json") production_streaming_qdrant_sharded_smoke_payload = _load_json(root / "benchmarks" / "production_streaming_load_qdrant_sharded_smoke_results.json") + production_streaming_qdrant_sharded_10m_payload = _load_json(root / "benchmarks" / "production_streaming_load_qdrant_sharded_10m_results.json") production_streaming_qdrant_10m_plan_payload = _load_json(root / "benchmarks" / "production_streaming_load_qdrant_10m_plan.json") production_streaming_qdrant_sharded_10m_plan_payload = _load_json(root / "benchmarks" / "production_streaming_load_qdrant_sharded_10m_plan.json") production_streaming_qdrant_sharded_100m_plan_payload = _load_json(root / "benchmarks" / "production_streaming_load_qdrant_sharded_100m_plan.json") @@ -633,6 +634,7 @@ def _implemented_entries(root: Path) -> list[dict[str, Any]]: **_prefixed_ann_results("1M Qdrant tuned", production_streaming_qdrant_1m_tuned_payload), **_prefixed_ann_results("10M Qdrant measured", production_streaming_qdrant_10m_payload), **_prefixed_ann_results("Qdrant sharded smoke", production_streaming_qdrant_sharded_smoke_payload), + **_prefixed_ann_results("10M Qdrant sharded measured", production_streaming_qdrant_sharded_10m_payload), **_streaming_plan_results("10M Qdrant preflight", production_streaming_qdrant_10m_plan_payload), **_streaming_plan_results("10M Qdrant sharded preflight", production_streaming_qdrant_sharded_10m_plan_payload), **_streaming_plan_results("100M Qdrant sharded preflight", production_streaming_qdrant_sharded_100m_plan_payload), @@ -1372,8 +1374,8 @@ def _implemented_entries(root: Path) -> list[dict[str, Any]]: "build_ms", ], "current": production_streaming_results, - "target": "Keep 10M compressed FAISS IVF-PQ above recall@10 0.95 and p99 below 100 ms, keep tuned 1M Qdrant streaming below p99 100 ms, keep Qdrant, sharded Qdrant, and pgvector streaming smokes green, keep 10M single-service Qdrant, 10M sharded Qdrant, 10M pgvector, 50M FAISS, and 100M sharded Qdrant preflights reproducible, then run those service profiles on hardware sized for the index.", - "next_step": "Run .github/workflows/production-streaming-load.yml with service credentials on a sized runner for 10M Qdrant, 10M sharded Qdrant, 10M pgvector, 50M compressed FAISS, and 100M sharded Qdrant profiles.", + "target": "Keep strict 10M single-service and four-service sharded Qdrant, 10M/50M compressed FAISS, and service smokes green while completing 10M pgvector and 100M sharded Qdrant evidence.", + "next_step": "Run .github/workflows/production-streaming-load.yml with sized pgvector infrastructure for 10M, then execute the 100M sharded Qdrant profile.", }, { "id": "scale_readiness", diff --git a/benchmarks/cluster_autoscale_results.json b/benchmarks/cluster_autoscale_results.json index a862e4e..cb7f3ab 100644 --- a/benchmarks/cluster_autoscale_results.json +++ b/benchmarks/cluster_autoscale_results.json @@ -1,7 +1,7 @@ { "schema": "wavemind.cluster_autoscale_report.v1", "generated_at": "2026-07-09T22:45:10Z", - "source_ref": "be2cebfd776b", + "source_ref": "c4f786e131c8", "source_file": "benchmarks/scale_readiness_results.json", "claim_boundary": "Cluster autoscale evidence is extracted from the checked-in scale-readiness artifact. It proves deterministic shard placement, failure-domain availability, autoscale planning, rebalance planning, operator reconciliation, quorum safety, active-active convergence, field-state CRDT behavior, and the 100M capacity envelope on these fixtures. It is not a real 100M vector-query latency benchmark, managed Kubernetes production run, or independent multi-region SLO.", "summary": { diff --git a/benchmarks/cost_efficiency_leaderboard.py b/benchmarks/cost_efficiency_leaderboard.py index 78b7c53..974aab9 100644 --- a/benchmarks/cost_efficiency_leaderboard.py +++ b/benchmarks/cost_efficiency_leaderboard.py @@ -27,6 +27,7 @@ "benchmarks/production_streaming_load_qdrant_1m_tuned_results.json", "benchmarks/production_streaming_load_qdrant_10m_results.json", "benchmarks/production_streaming_load_qdrant_sharded_smoke_results.json", + "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", "benchmarks/production_streaming_load_pgvector_smoke_results.json", ) diff --git a/benchmarks/cost_efficiency_results.json b/benchmarks/cost_efficiency_results.json index 32fb4d1..9f2b7d5 100644 --- a/benchmarks/cost_efficiency_results.json +++ b/benchmarks/cost_efficiency_results.json @@ -1,13 +1,13 @@ { "schema": "wavemind.cost_efficiency_leaderboard.v1", - "generated_at": "2026-07-11T03:37:56Z", - "source_ref": "be2cebfd776b", + "generated_at": "2026-07-11T05:56:44Z", + "source_ref": "c4f786e131c8", "claim_boundary": "Measured rows come from checked-in load artifacts. Planned rows are capacity and cost contracts only; they do not unlock production latency or recall claims until the matching benchmark result exists.", "summary": { - "measured_row_count": 21, + "measured_row_count": 22, "planned_row_count": 5, "measured_slo_pass_count": 9, - "measured_valid_cost_count": 12, + "measured_valid_cost_count": 13, "planned_valid_cost_count": 5, "measured_frontier_profiles": [ "sub_100k-wavemind-numpy-streaming-production_streaming_load_smoke_results", @@ -15,6 +15,7 @@ "1m-qdrant-service-streaming-production_streaming_load_qdrant_1m_tuned_results", "1m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_1m_results", "100k-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_100k_results", + "10m-qdrant-sharded-service-streaming-production_streaming_load_qdrant_sharded_10m_results", "10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results", "10m-qdrant-service-streaming-production_streaming_load_qdrant_10m_results" ], @@ -24,7 +25,7 @@ ], "best_measured_by_target_class": { "100k": "100k-qdrant-service-production_load_qdrant_100k_tuned_results", - "10m": "10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results", + "10m": "10m-qdrant-sharded-service-streaming-production_streaming_load_qdrant_sharded_10m_results", "1m": "1m-qdrant-service-streaming-production_streaming_load_qdrant_1m_tuned_results", "sub_100k": "sub_100k-wavemind-numpy-streaming-production_streaming_load_smoke_results" }, @@ -142,6 +143,30 @@ } ], "10m": [ + { + "profile": "10m-qdrant-sharded-service-streaming-production_streaming_load_qdrant_sharded_10m_results", + "evidence_level": "measured", + "source_file": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", + "engine": "Qdrant sharded service streaming", + "memory_count": 10000000, + "vector_dim": 128, + "target_class": "10m", + "recall_at_k": 0.9925, + "p99_latency_ms": 71.27849999233149, + "avg_latency_ms": 46.60563879973779, + "target_qps": null, + "slo_status": "scale_required", + "cost_status": "valid_slo", + "compute_cost_per_1m_queries_usd": 4.722222222222222, + "monthly_total_cost_at_target_qps_usd": 3104.8841857910156, + "monthly_total_cost_per_1m_memories_usd": 310.48841857910156, + "estimated_storage_gb": 23.84185791015625, + "claim_status": "measured_scale_required", + "slo_pass": false, + "valid_cost": true, + "frontier_score": 29486.657352616447, + "rank": 1 + }, { "profile": "10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results", "evidence_level": "measured", @@ -164,7 +189,7 @@ "slo_pass": false, "valid_cost": true, "frontier_score": 33869.867549553164, - "rank": 1 + "rank": 2 }, { "profile": "10m-qdrant-service-streaming-production_streaming_load_qdrant_10m_results", @@ -188,7 +213,7 @@ "slo_pass": false, "valid_cost": true, "frontier_score": 64899.94591989463, - "rank": 2 + "rank": 3 } ], "1m": [ @@ -950,6 +975,30 @@ "frontier_score": 4158.371581552904, "rank": 10 }, + { + "profile": "10m-qdrant-sharded-service-streaming-production_streaming_load_qdrant_sharded_10m_results", + "evidence_level": "measured", + "source_file": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", + "engine": "Qdrant sharded service streaming", + "memory_count": 10000000, + "vector_dim": 128, + "target_class": "10m", + "recall_at_k": 0.9925, + "p99_latency_ms": 71.27849999233149, + "avg_latency_ms": 46.60563879973779, + "target_qps": null, + "slo_status": "scale_required", + "cost_status": "valid_slo", + "compute_cost_per_1m_queries_usd": 4.722222222222222, + "monthly_total_cost_at_target_qps_usd": 3104.8841857910156, + "monthly_total_cost_per_1m_memories_usd": 310.48841857910156, + "estimated_storage_gb": 23.84185791015625, + "claim_status": "measured_scale_required", + "slo_pass": false, + "valid_cost": true, + "frontier_score": 29486.657352616447, + "rank": 11 + }, { "profile": "10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results", "evidence_level": "measured", @@ -972,7 +1021,7 @@ "slo_pass": false, "valid_cost": true, "frontier_score": 33869.867549553164, - "rank": 11 + "rank": 12 }, { "profile": "10m-qdrant-service-streaming-production_streaming_load_qdrant_10m_results", @@ -996,7 +1045,7 @@ "slo_pass": false, "valid_cost": true, "frontier_score": 64899.94591989463, - "rank": 12 + "rank": 13 }, { "profile": "1m-qdrant-service-streaming-production_streaming_load_qdrant_1m_results", @@ -1020,7 +1069,7 @@ "slo_pass": false, "valid_cost": false, "frontier_score": 19.708175013842546, - "rank": 13 + "rank": 14 }, { "profile": "1m-qdrant-service-production_load_qdrant_1m_tuned_results", @@ -1044,7 +1093,7 @@ "slo_pass": false, "valid_cost": false, "frontier_score": 856.5459610857889, - "rank": 14 + "rank": 15 }, { "profile": "1m-qdrant-service-production_load_qdrant_1m_ef_sweep_results", @@ -1068,7 +1117,7 @@ "slo_pass": false, "valid_cost": false, "frontier_score": 1355.2697384743992, - "rank": 15 + "rank": 16 }, { "profile": "1m-qdrant-service-production_load_qdrant_1m_ef_sweep_results", @@ -1092,7 +1141,7 @@ "slo_pass": false, "valid_cost": false, "frontier_score": 1127.117419200782, - "rank": 16 + "rank": 17 }, { "profile": "1m-qdrant-service-production_load_qdrant_1m_ef_sweep_results", @@ -1116,7 +1165,7 @@ "slo_pass": false, "valid_cost": false, "frontier_score": 1652.2388029878396, - "rank": 17 + "rank": 18 }, { "profile": "1m-qdrant-service-production_load_qdrant_1m_ef_sweep_results", @@ -1140,7 +1189,7 @@ "slo_pass": false, "valid_cost": false, "frontier_score": 2506.808217106415, - "rank": 18 + "rank": 19 }, { "profile": "1m-qdrant-service-production_load_qdrant_1m_ef_sweep_results", @@ -1164,7 +1213,7 @@ "slo_pass": false, "valid_cost": false, "frontier_score": 2261.7523882657606, - "rank": 19 + "rank": 20 }, { "profile": "100k-wavemind-pgvector-production_load_results", @@ -1188,7 +1237,7 @@ "slo_pass": false, "valid_cost": false, "frontier_score": 0.035328000000000005, - "rank": 20 + "rank": 21 }, { "profile": "1m-qdrant-service-production_load_qdrant_1m_results", @@ -1212,7 +1261,7 @@ "slo_pass": false, "valid_cost": false, "frontier_score": 0.10409142857142857, - "rank": 21 + "rank": 22 } ], "planned_rows": [ diff --git a/benchmarks/memory_os_intelligence_results.json b/benchmarks/memory_os_intelligence_results.json index 15107dc..fdbdf08 100644 --- a/benchmarks/memory_os_intelligence_results.json +++ b/benchmarks/memory_os_intelligence_results.json @@ -1,7 +1,7 @@ { "schema": "wavemind.memory_os_intelligence_report.v1", "generated_at": "2026-07-09T22:45:10Z", - "source_ref": "be2cebfd776b", + "source_ref": "c4f786e131c8", "source_files": [ "benchmarks/scale_readiness_results.json", "benchmarks/agent_coherence_results.json", diff --git a/benchmarks/production_evidence_bundle_results.json b/benchmarks/production_evidence_bundle_results.json index c94e919..60a2706 100644 --- a/benchmarks/production_evidence_bundle_results.json +++ b/benchmarks/production_evidence_bundle_results.json @@ -1,11 +1,11 @@ { "schema": "wavemind.production_evidence_bundle.v1", - "generated_at": "2026-07-11T03:38:02Z", + "generated_at": "2026-07-11T05:56:51Z", "claim_status": "claims_limited", "summary": { "claim_status": "claims_limited", "strict_overall_status": "action_required", - "strict_pass_count": 3, + "strict_pass_count": 4, "strict_total_requirements": 8, "preflight_overall_status": "action_required", "preflight_ready_count": 0, @@ -17,16 +17,16 @@ "production_scale_run_contract_status": "available", "production_scale_run_profile_count": 5, "production_scale_run_target_memories_total": 180000000, - "next_action_count": 5 + "next_action_count": 4 }, "strict_production_evidence": { "schema": "wavemind.production_evidence.v1", - "generated_at": "2026-07-11T03:38:02Z", + "generated_at": "2026-07-11T05:56:51Z", "overall_status": "action_required", "summary": { "overall_status": "action_required", - "pass_count": 3, - "action_required_count": 5, + "pass_count": 4, + "action_required_count": 4, "fail_count": 0, "total_requirements": 8 }, @@ -78,14 +78,12 @@ { "id": "qdrant_sharded_10m_service", "title": "10M sharded Qdrant service load", - "status": "action_required", - "evidence": "no checked-in 10,000,000-vector result for Qdrant sharded service streaming", + "status": "pass", + "evidence": "Qdrant sharded service streaming: vectors 10000000, recall 0.9925, p99 71.27849999233149 ms, cost valid_slo, source 11d0be8c62e8, run local-qdrant-sharded10m-4x-v1182-20260711", "artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", "command": "python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json", "claim_unlocked": "Horizontally sharded Qdrant service recall/latency SLO.", - "issues": [ - "missing artifact" - ] + "issues": [] }, { "id": "pgvector_10m_service", @@ -125,7 +123,7 @@ }, "production_evidence_preflight": { "schema": "wavemind.production_evidence_preflight.v1", - "generated_at": "2026-07-11T03:38:02Z", + "generated_at": "2026-07-11T05:56:51Z", "overall_status": "action_required", "summary": { "overall_status": "action_required", @@ -303,7 +301,7 @@ }, "production_readiness": { "schema": "wavemind.production_readiness.v1", - "generated_at": "2026-07-11T03:37:59Z", + "generated_at": "2026-07-11T05:56:48Z", "overall_status": "pass", "readiness_score": 1.0, "summary": { @@ -320,7 +318,7 @@ "title": "Checked-in benchmark artifacts are synchronized", "status": "pass", "requirement": "Benchmark matrix, report, and leaderboard must render from the same checked-in JSON.", - "evidence": "audit status pass, generated_at 2026-07-11T03:28:43Z", + "evidence": "audit status pass, generated_at 2026-07-11T05:53:28Z", "next_step": "Keep the benchmark refresh workflow green and block stale artifacts before release." }, { @@ -655,13 +653,13 @@ "artifact_audit": { "schema": "wavemind.benchmark_artifact_audit.v1", "status": "pass", - "checked_at": "2026-07-11T03:28:53Z", - "generated_at": "2026-07-11T03:28:43Z", - "source_ref": "be2cebfd776b", + "checked_at": "2026-07-11T05:53:37Z", + "generated_at": "2026-07-11T05:53:28Z", + "source_ref": "c4f786e131c8", "workflow_run_id": null, "refresh_profile": "local", "max_age_days": 8.0, - "age_days": 0.00011505136574074075, + "age_days": 0.00011299333333333334, "implemented_count": 37, "runner_ready_count": 3, "planned_count": 6, @@ -804,23 +802,6 @@ "command": "gh workflow run serverless-observed-telemetry.yml -f nodes=\"$WAVEMIND_SERVERLESS_NODES\" -f seed_mode=first -f commit_results=true", "claim_unlocked": "Hosted/serverless p99, cold-start, error-rate, and scale-out SLO." }, - { - "id": "qdrant_sharded_10m_service", - "title": "10M sharded Qdrant service load", - "strict_status": "action_required", - "preflight_status": "action_required", - "artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", - "output_artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", - "issues": [ - "missing artifact" - ], - "missing_env": [ - "WAVEMIND_QDRANT_URLS" - ], - "warnings": [], - "command": "python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json", - "claim_unlocked": "Horizontally sharded Qdrant service recall/latency SLO." - }, { "id": "pgvector_10m_service", "title": "10M pgvector service load", diff --git a/benchmarks/production_evidence_dispatch_results.json b/benchmarks/production_evidence_dispatch_results.json index d9647e2..543c3b3 100644 --- a/benchmarks/production_evidence_dispatch_results.json +++ b/benchmarks/production_evidence_dispatch_results.json @@ -1,13 +1,13 @@ { "schema": "wavemind.production_evidence_dispatch.v1", - "generated_at": "2026-07-11T03:38:02Z", + "generated_at": "2026-07-11T05:56:51Z", "overall_status": "action_required", "summary": { "overall_status": "action_required", "total_jobs": 8, "ready_to_dispatch_count": 0, - "blocked_by_preflight_count": 5, - "complete_count": 3, + "blocked_by_preflight_count": 4, + "complete_count": 4, "commit_results_default": false, "runner_label": "self-hosted-large", "wave_counts": { @@ -17,8 +17,8 @@ "service-scale-10m": 3 }, "status_counts": { - "blocked_by_preflight": 5, - "complete": 3 + "blocked_by_preflight": 4, + "complete": 4 } }, "launch_policy": { @@ -228,10 +228,10 @@ { "id": "qdrant_sharded_10m_service", "title": "10M sharded Qdrant service load", - "status": "blocked_by_preflight", - "dispatch_required": true, + "status": "complete", + "dispatch_required": false, "ready": false, - "strict_status": "action_required", + "strict_status": "pass", "preflight_status": "action_required", "wave": "service-scale-10m", "workflow": "production-streaming-load.yml", diff --git a/benchmarks/production_evidence_preflight_results.json b/benchmarks/production_evidence_preflight_results.json index 8ee9d5c..c95328f 100644 --- a/benchmarks/production_evidence_preflight_results.json +++ b/benchmarks/production_evidence_preflight_results.json @@ -1,6 +1,6 @@ { "schema": "wavemind.production_evidence_preflight.v1", - "generated_at": "2026-07-11T03:38:01Z", + "generated_at": "2026-07-11T05:56:50Z", "overall_status": "action_required", "summary": { "overall_status": "action_required", diff --git a/benchmarks/production_evidence_results.json b/benchmarks/production_evidence_results.json index daad81a..3427082 100644 --- a/benchmarks/production_evidence_results.json +++ b/benchmarks/production_evidence_results.json @@ -1,11 +1,11 @@ { "schema": "wavemind.production_evidence.v1", - "generated_at": "2026-07-11T03:38:00Z", + "generated_at": "2026-07-11T05:56:49Z", "overall_status": "action_required", "summary": { "overall_status": "action_required", - "pass_count": 3, - "action_required_count": 5, + "pass_count": 4, + "action_required_count": 4, "fail_count": 0, "total_requirements": 8 }, @@ -57,14 +57,12 @@ { "id": "qdrant_sharded_10m_service", "title": "10M sharded Qdrant service load", - "status": "action_required", - "evidence": "no checked-in 10,000,000-vector result for Qdrant sharded service streaming", + "status": "pass", + "evidence": "Qdrant sharded service streaming: vectors 10000000, recall 0.9925, p99 71.27849999233149 ms, cost valid_slo, source 11d0be8c62e8, run local-qdrant-sharded10m-4x-v1182-20260711", "artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", "command": "python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json", "claim_unlocked": "Horizontally sharded Qdrant service recall/latency SLO.", - "issues": [ - "missing artifact" - ] + "issues": [] }, { "id": "pgvector_10m_service", diff --git a/benchmarks/production_readiness_results.json b/benchmarks/production_readiness_results.json index e1a1e8c..f02e0c0 100644 --- a/benchmarks/production_readiness_results.json +++ b/benchmarks/production_readiness_results.json @@ -1,6 +1,6 @@ { "schema": "wavemind.production_readiness.v1", - "generated_at": "2026-07-11T03:37:59Z", + "generated_at": "2026-07-11T05:56:48Z", "overall_status": "pass", "readiness_score": 1.0, "summary": { @@ -17,7 +17,7 @@ "title": "Checked-in benchmark artifacts are synchronized", "status": "pass", "requirement": "Benchmark matrix, report, and leaderboard must render from the same checked-in JSON.", - "evidence": "audit status pass, generated_at 2026-07-11T03:28:43Z", + "evidence": "audit status pass, generated_at 2026-07-11T05:53:28Z", "next_step": "Keep the benchmark refresh workflow green and block stale artifacts before release." }, { diff --git a/benchmarks/production_streaming_load_benchmark.py b/benchmarks/production_streaming_load_benchmark.py index f64f98c..7acc49f 100644 --- a/benchmarks/production_streaming_load_benchmark.py +++ b/benchmarks/production_streaming_load_benchmark.py @@ -375,6 +375,7 @@ class QdrantShardTarget: url: str collection_name: str api_key: str | None = None + grpc_port: int | None = None def _qdrant_shard_index(point_id: int, shard_count: int) -> int: @@ -391,15 +392,21 @@ def _qdrant_shard_targets_from_env(base_collection_name: str) -> list[QdrantShar urls = [url] api_keys = _split_env_list(os.environ.get("WAVEMIND_QDRANT_API_KEYS")) default_api_key = os.environ.get("WAVEMIND_QDRANT_API_KEY") + grpc_ports = _split_env_list(os.environ.get("WAVEMIND_QDRANT_GRPC_PORTS")) + default_grpc_port = _optional_int_env("WAVEMIND_QDRANT_GRPC_PORT") targets: list[QdrantShardTarget] = [] for index, url in enumerate(urls): api_key = api_keys[index] if index < len(api_keys) else default_api_key + grpc_port = ( + int(grpc_ports[index]) if index < len(grpc_ports) else default_grpc_port + ) targets.append( QdrantShardTarget( index=index, url=url, collection_name=f"{base_collection_name}_s{index:03d}", api_key=api_key, + grpc_port=grpc_port, ) ) return targets @@ -484,26 +491,43 @@ def _iter_qdrant_point_chunks( ] +def _iter_qdrant_shard_point_chunks( + ids: np.ndarray, + vectors: np.ndarray, + *, + shard_index: int, + shard_count: int, + point_type: Any, + chunk_size: int, +) -> Iterable[list[Any]]: + if chunk_size <= 0: + raise ValueError("chunk size must be positive") + point_chunk: list[Any] = [] + for point_id, vector in zip(ids, vectors): + if _qdrant_shard_index(int(point_id), shard_count) != shard_index: + continue + point_chunk.append(point_type(id=int(point_id), vector=vector.tolist())) + if len(point_chunk) >= chunk_size: + yield point_chunk + point_chunk = [] + if point_chunk: + yield point_chunk + + def _upsert_qdrant_shards( *, executor: concurrent.futures.Executor, clients: list[Any], targets: list[QdrantShardTarget], - points_by_shard: dict[int, list[Any]], - upsert_batch_size: int, + point_chunks_by_shard: dict[int, Iterable[list[Any]]], ) -> int: - active_shards = [ - shard_index - for shard_index, points in points_by_shard.items() - if points - ] + active_shards = list(point_chunks_by_shard) def upsert_shard(shard_index: int) -> int: - points = points_by_shard[shard_index] client = clients[shard_index] collection_name = targets[shard_index].collection_name inserted = 0 - for point_chunk in _chunks(points, upsert_batch_size): + for point_chunk in point_chunks_by_shard[shard_index]: client.upsert(collection_name=collection_name, points=point_chunk) inserted += len(point_chunk) return inserted @@ -958,10 +982,17 @@ def _streaming_plan_row( "WAVEMIND_QDRANT_COLLECTION_PREFIX": "wavemind_streaming_load_10m", "WAVEMIND_QDRANT_UPSERT_BATCH_SIZE": "2000", "WAVEMIND_QDRANT_FANOUT_WORKERS": str(shard_count), + "WAVEMIND_QDRANT_PREFER_GRPC": "1", + "WAVEMIND_QDRANT_GRPC_PORT": "6334", + "WAVEMIND_QDRANT_QUERY_TIMEOUT_SECONDS": "60", "WAVEMIND_QDRANT_WAIT_AFTER_BUILD_SECONDS": "30", "WAVEMIND_QDRANT_WARMUP_QUERIES": "100", "WAVEMIND_QDRANT_VECTOR_ON_DISK": "1", - "WAVEMIND_QDRANT_HNSW_ON_DISK": "1", + "WAVEMIND_QDRANT_HNSW_ON_DISK": "0", + "WAVEMIND_QDRANT_SCALAR_QUANTIZATION": "1", + "WAVEMIND_QDRANT_SCALAR_QUANTILE": "0.99", + "WAVEMIND_QDRANT_SCALAR_ALWAYS_RAM": "1", + "WAVEMIND_QDRANT_QUANTIZATION_RESCORE": "0", "WAVEMIND_QDRANT_INDEX_READY_TIMEOUT_SECONDS": "1800", "WAVEMIND_QDRANT_REQUIRE_FULL_INDEX": "1", "WAVEMIND_QDRANT_DEFER_INDEXING": "1", @@ -1990,6 +2021,10 @@ def run_qdrant_sharded_streaming( HnswConfigDiff, OptimizersConfigDiff, PointStruct, + QuantizationSearchParams, + ScalarQuantization, + ScalarQuantizationConfig, + ScalarType, SearchParams, VectorParams, ) @@ -2032,7 +2067,10 @@ def run_qdrant_sharded_streaming( ) checkpoint_metadata["collection_prefix"] = base_collection_name _write_checkpoint(checkpoint_path, checkpoint) - targets = _qdrant_shard_targets_from_env(base_collection_name) + try: + targets = _qdrant_shard_targets_from_env(base_collection_name) + except ValueError as exc: + return skipped_result(engine, f"Invalid Qdrant shard transport configuration: {exc}") if len(targets) < 2: return skipped_result( engine, @@ -2050,6 +2088,18 @@ def run_qdrant_sharded_streaming( if collection_config["hnsw"] else None ) + scalar_quantization_config = collection_config["scalar_quantization"] + quantization_config = ( + ScalarQuantization( + scalar=ScalarQuantizationConfig( + type=ScalarType.INT8, + quantile=float(scalar_quantization_config["quantile"]), + always_ram=bool(scalar_quantization_config["always_ram"]), + ) + ) + if scalar_quantization_config + else None + ) ingest_optimizers = dict(collection_config["optimizers"]) if deferred_indexing["enabled"]: ingest_optimizers["indexing_threshold"] = deferred_indexing[ @@ -2062,6 +2112,7 @@ def run_qdrant_sharded_streaming( { "hnsw_config": hnsw_config, "optimizers_config": ingest_optimizers_config, + "quantization_config": quantization_config, "on_disk_payload": collection_config["on_disk_payload"], "shard_number": collection_config["shard_number"], } @@ -2073,6 +2124,8 @@ def run_qdrant_sharded_streaming( QdrantClient( url=target.url, api_key=target.api_key, + grpc_port=target.grpc_port, + prefer_grpc=_bool_env("WAVEMIND_QDRANT_PREFER_GRPC", False), timeout=float(os.environ.get("WAVEMIND_QDRANT_TIMEOUT", "120")), ) ) @@ -2085,6 +2138,7 @@ def query_target(client: Any, target: QdrantShardTarget, query: np.ndarray, sear limit=top_k, with_payload=False, search_params=search_params, + **query_kwargs, ).points ) @@ -2092,6 +2146,12 @@ def query_target(client: Any, target: QdrantShardTarget, query: np.ndarray, sear source_ids = choose_source_ids(count, query_count, seed) source_vectors: dict[int, np.ndarray] = _checkpoint_source_vectors(checkpoint) completed_batches = _checkpoint_completed_batches(checkpoint) + complete_resume = _checkpoint_complete_for_run( + checkpoint, + count=count, + batch_size=batch_size, + source_ids=source_ids, + ) started = time.perf_counter() if not completed_batches: for client, target in zip(clients, targets): @@ -2105,56 +2165,69 @@ def query_target(client: Any, target: QdrantShardTarget, query: np.ndarray, sear timeout=int(os.environ.get("WAVEMIND_QDRANT_COLLECTION_TIMEOUT", "120")), **recreate_kwargs, ) - elif deferred_indexing["enabled"]: + elif complete_resume and ( + hnsw_config is not None or quantization_config is not None + ): + for client, target in zip(clients, targets): + client.update_collection( + collection_name=target.collection_name, + hnsw_config=hnsw_config, + quantization_config=quantization_config, + timeout=int( + os.environ.get("WAVEMIND_QDRANT_COLLECTION_TIMEOUT", "120") + ), + ) + elif deferred_indexing["enabled"] and not complete_resume: for client, target in zip(clients, targets): client.update_collection( collection_name=target.collection_name, optimizers_config=ingest_optimizers_config, timeout=int(os.environ.get("WAVEMIND_QDRANT_COLLECTION_TIMEOUT", "120")), ) - with concurrent.futures.ThreadPoolExecutor( - max_workers=fanout_workers - ) as ingest_executor: - for ids, vectors, captured in iter_vector_batches( - count=count, - dim=dim, - seed=seed + count, - batch_size=batch_size, - source_ids=source_ids, - ): - batch_start = int(ids[0]) if len(ids) else 0 - if batch_start not in completed_batches: - points_by_shard: dict[int, list[Any]] = { - index: [] for index in range(len(targets)) - } - for point_id, vector in zip(ids, vectors): - shard_index = _qdrant_shard_index( - int(point_id), len(targets) - ) - points_by_shard[shard_index].append( - PointStruct(id=int(point_id), vector=vector.tolist()) + if not complete_resume: + with concurrent.futures.ThreadPoolExecutor( + max_workers=fanout_workers + ) as ingest_executor: + for ids, vectors, captured in iter_vector_batches( + count=count, + dim=dim, + seed=seed + count, + batch_size=batch_size, + source_ids=source_ids, + ): + batch_start = int(ids[0]) if len(ids) else 0 + if batch_start not in completed_batches: + point_chunks_by_shard = { + target.index: _iter_qdrant_shard_point_chunks( + ids, + vectors, + shard_index=target.index, + shard_count=len(targets), + point_type=PointStruct, + chunk_size=upsert_batch_size, + ) + for target in targets + } + inserted = _upsert_qdrant_shards( + executor=ingest_executor, + clients=clients, + targets=targets, + point_chunks_by_shard=point_chunks_by_shard, ) - inserted = _upsert_qdrant_shards( - executor=ingest_executor, - clients=clients, - targets=targets, - points_by_shard=points_by_shard, - upsert_batch_size=upsert_batch_size, + if inserted != len(ids): + raise RuntimeError( + "Qdrant sharded upsert did not acknowledge every point" + ) + source_vectors.update(captured) + _record_checkpoint_batch( + path=checkpoint_path, + payload=checkpoint, + batch_start=batch_start, + captured=captured, ) - if inserted != len(ids): - raise RuntimeError( - "Qdrant sharded upsert did not acknowledge every point" - ) - source_vectors.update(captured) - _record_checkpoint_batch( - path=checkpoint_path, - payload=checkpoint, - batch_start=batch_start, - captured=captured, - ) - completed_batches.add(batch_start) + completed_batches.add(batch_start) index_restore_ms = 0.0 - if deferred_indexing["enabled"]: + if deferred_indexing["enabled"] and not complete_resume: restore_started = time.perf_counter() final_optimizers = dict(collection_config["optimizers"]) final_optimizers["indexing_threshold"] = deferred_indexing[ @@ -2202,12 +2275,30 @@ def query_target(client: Any, target: QdrantShardTarget, query: np.ndarray, sear "yes", "on", } + quantization_search_params = None + if scalar_quantization_config: + quantization_search_params = QuantizationSearchParams( + ignore=_optional_bool_env( + "WAVEMIND_QDRANT_QUANTIZATION_IGNORE" + ), + rescore=_optional_bool_env( + "WAVEMIND_QDRANT_QUANTIZATION_RESCORE" + ), + oversampling=_optional_float_env( + "WAVEMIND_QDRANT_QUANTIZATION_OVERSAMPLING" + ), + ) search_params = None - if hnsw_ef or exact: + if hnsw_ef or exact or quantization_search_params is not None: search_params = SearchParams( hnsw_ef=int(hnsw_ef) if hnsw_ef else None, exact=exact or None, + quantization=quantization_search_params, ) + query_timeout_seconds = _optional_int_env( + "WAVEMIND_QDRANT_QUERY_TIMEOUT_SECONDS" + ) + query_kwargs = _without_none({"timeout": query_timeout_seconds}) queries = make_queries(source_ids=source_ids, source_vectors=source_vectors, seed=seed + count, noise=noise) with concurrent.futures.ThreadPoolExecutor(max_workers=fanout_workers) as executor: warmup_queries = int(os.environ.get("WAVEMIND_QDRANT_WARMUP_QUERIES", "0")) @@ -2263,9 +2354,29 @@ def query_target(client: Any, target: QdrantShardTarget, query: np.ndarray, sear "search_params": { "hnsw_ef": int(hnsw_ef) if hnsw_ef else None, "exact": exact, + "quantization": { + "enabled": quantization_search_params is not None, + "ignore": _optional_bool_env( + "WAVEMIND_QDRANT_QUANTIZATION_IGNORE" + ), + "rescore": _optional_bool_env( + "WAVEMIND_QDRANT_QUANTIZATION_RESCORE" + ), + "oversampling": _optional_float_env( + "WAVEMIND_QDRANT_QUANTIZATION_OVERSAMPLING" + ), + }, + }, + "transport": { + "prefer_grpc": _bool_env( + "WAVEMIND_QDRANT_PREFER_GRPC", False + ), + "grpc_ports": [target.grpc_port for target in targets], + "query_timeout_seconds": query_timeout_seconds, }, "collection_params": collection_config, "upsert_batch_size": upsert_batch_size, + "qdrant_checkpoint_complete_resume": bool(complete_resume), "memory_mode": "horizontally sharded streaming upsert; parallel fanout query merge", **_checkpoint_extra(checkpoint_path, checkpoint, completed_batches), }, diff --git a/benchmarks/production_streaming_load_qdrant_sharded_100m_plan.json b/benchmarks/production_streaming_load_qdrant_sharded_100m_plan.json index 7271770..082e6d0 100644 --- a/benchmarks/production_streaming_load_qdrant_sharded_100m_plan.json +++ b/benchmarks/production_streaming_load_qdrant_sharded_100m_plan.json @@ -25,7 +25,7 @@ "plan_only": true, "runner_storage_root": "state/production-runs", "disk_free_path": "C:/work/particles/wavemind 2.0/state", - "disk_free_gb": 0.0 + "disk_free_gb": 10.771 }, "preflight": { "python": "3.13.3", @@ -35,9 +35,9 @@ "server_version": "28.3.0" }, "modules": { - "faiss": true, + "faiss": false, "qdrant_client": true, - "psycopg": true, + "psycopg": false, "numpy": true }, "environment": { @@ -67,7 +67,7 @@ }, "disk": { "root": "C:\\", - "free_gb": 2.52, + "free_gb": 10.77, "total_gb": 231.85 } }, @@ -86,7 +86,7 @@ "estimated_float_vector_storage_gb": 47.684, "estimated_application_storage_gb": 238.419, "required_local_free_gb": 0.009, - "disk_free_gb": 0.0, + "disk_free_gb": 10.771, "runner_storage_root": "state/production-runs", "disk_free_path": "C:/work/particles/wavemind 2.0/state", "safety_factor": 1.25, @@ -104,10 +104,17 @@ "WAVEMIND_QDRANT_COLLECTION_PREFIX": "wavemind_streaming_load_10m", "WAVEMIND_QDRANT_UPSERT_BATCH_SIZE": "2000", "WAVEMIND_QDRANT_FANOUT_WORKERS": "8", + "WAVEMIND_QDRANT_PREFER_GRPC": "1", + "WAVEMIND_QDRANT_GRPC_PORT": "6334", + "WAVEMIND_QDRANT_QUERY_TIMEOUT_SECONDS": "60", "WAVEMIND_QDRANT_WAIT_AFTER_BUILD_SECONDS": "30", "WAVEMIND_QDRANT_WARMUP_QUERIES": "100", "WAVEMIND_QDRANT_VECTOR_ON_DISK": "1", - "WAVEMIND_QDRANT_HNSW_ON_DISK": "1", + "WAVEMIND_QDRANT_HNSW_ON_DISK": "0", + "WAVEMIND_QDRANT_SCALAR_QUANTIZATION": "1", + "WAVEMIND_QDRANT_SCALAR_QUANTILE": "0.99", + "WAVEMIND_QDRANT_SCALAR_ALWAYS_RAM": "1", + "WAVEMIND_QDRANT_QUANTIZATION_RESCORE": "0", "WAVEMIND_QDRANT_INDEX_READY_TIMEOUT_SECONDS": "1800", "WAVEMIND_QDRANT_REQUIRE_FULL_INDEX": "1", "WAVEMIND_QDRANT_DEFER_INDEXING": "1", @@ -119,8 +126,7 @@ "command": "python benchmarks/production_streaming_load_benchmark.py --sizes 100000000 --dim 128 --queries 5000 --top-k 10 --seed 42 --noise 0.08 --batch-size 10000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 500.0 --replicas 8 --autoscaling-max-replicas 128 --capacity-headroom 0.7 --output benchmarks\\production_streaming_load_qdrant_sharded_100m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-100000000.checkpoint.json", "status": "action_required", "blockers": [ - "missing_env:WAVEMIND_QDRANT_URLS", - "insufficient_local_disk_for_index_and_transient_batches" + "missing_env:WAVEMIND_QDRANT_URLS" ], "claim_boundary": "preflight only; this is not a completed latency or recall benchmark" } diff --git a/benchmarks/production_streaming_load_qdrant_sharded_10m_plan.json b/benchmarks/production_streaming_load_qdrant_sharded_10m_plan.json index 135928c..3fef226 100644 --- a/benchmarks/production_streaming_load_qdrant_sharded_10m_plan.json +++ b/benchmarks/production_streaming_load_qdrant_sharded_10m_plan.json @@ -25,7 +25,7 @@ "plan_only": true, "runner_storage_root": "state/production-runs", "disk_free_path": "C:/work/particles/wavemind 2.0/state", - "disk_free_gb": 0.0 + "disk_free_gb": 10.771 }, "preflight": { "python": "3.13.3", @@ -35,9 +35,9 @@ "server_version": "28.3.0" }, "modules": { - "faiss": true, + "faiss": false, "qdrant_client": true, - "psycopg": true, + "psycopg": false, "numpy": true }, "environment": { @@ -67,7 +67,7 @@ }, "disk": { "root": "C:\\", - "free_gb": 2.51, + "free_gb": 10.77, "total_gb": 231.85 } }, @@ -86,7 +86,7 @@ "estimated_float_vector_storage_gb": 4.768, "estimated_application_storage_gb": 23.842, "required_local_free_gb": 0.004, - "disk_free_gb": 0.0, + "disk_free_gb": 10.771, "runner_storage_root": "state/production-runs", "disk_free_path": "C:/work/particles/wavemind 2.0/state", "safety_factor": 1.25, @@ -104,10 +104,17 @@ "WAVEMIND_QDRANT_COLLECTION_PREFIX": "wavemind_streaming_load_10m", "WAVEMIND_QDRANT_UPSERT_BATCH_SIZE": "2000", "WAVEMIND_QDRANT_FANOUT_WORKERS": "4", + "WAVEMIND_QDRANT_PREFER_GRPC": "1", + "WAVEMIND_QDRANT_GRPC_PORT": "6334", + "WAVEMIND_QDRANT_QUERY_TIMEOUT_SECONDS": "60", "WAVEMIND_QDRANT_WAIT_AFTER_BUILD_SECONDS": "30", "WAVEMIND_QDRANT_WARMUP_QUERIES": "100", "WAVEMIND_QDRANT_VECTOR_ON_DISK": "1", - "WAVEMIND_QDRANT_HNSW_ON_DISK": "1", + "WAVEMIND_QDRANT_HNSW_ON_DISK": "0", + "WAVEMIND_QDRANT_SCALAR_QUANTIZATION": "1", + "WAVEMIND_QDRANT_SCALAR_QUANTILE": "0.99", + "WAVEMIND_QDRANT_SCALAR_ALWAYS_RAM": "1", + "WAVEMIND_QDRANT_QUANTIZATION_RESCORE": "0", "WAVEMIND_QDRANT_INDEX_READY_TIMEOUT_SECONDS": "1800", "WAVEMIND_QDRANT_REQUIRE_FULL_INDEX": "1", "WAVEMIND_QDRANT_DEFER_INDEXING": "1", @@ -119,8 +126,7 @@ "command": "python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks\\production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json", "status": "action_required", "blockers": [ - "missing_env:WAVEMIND_QDRANT_URLS", - "insufficient_local_disk_for_index_and_transient_batches" + "missing_env:WAVEMIND_QDRANT_URLS" ], "claim_boundary": "preflight only; this is not a completed latency or recall benchmark" } diff --git a/benchmarks/production_streaming_load_qdrant_sharded_10m_results.json b/benchmarks/production_streaming_load_qdrant_sharded_10m_results.json new file mode 100644 index 0000000..17fe7d9 --- /dev/null +++ b/benchmarks/production_streaming_load_qdrant_sharded_10m_results.json @@ -0,0 +1,304 @@ +{ + "schema": "wavemind.production_streaming_load.v1", + "generated_at": "2026-07-11T04:18:28Z", + "source_ref": "11d0be8c62e8678f5f154585f89eb4d016183cf7", + "execution_id": "local-qdrant-sharded10m-4x-v1182-20260711", + "execution_environment": "qdrant-1.18.2-docker-wsl2-four-posix-named-volumes-grpc-int8-ram", + "evidence_source": "local-service", + "workflow_run_id": null, + "workflow_run_url": null, + "scenario": { + "name": "production_streaming_load_profile", + "description": "Memory-bounded production load profile for 10M+ vector runs. Vectors are generated and inserted in batches. Quality is measured as target-recall: whether a noisy copy of a known source vector returns that source id in top-k. This is scalable to large N and complements exact-neighbor benchmarks at smaller N. Use persisted FAISS IVF-PQ for memory-bounded 10M+ compressed-index profiles.", + "sizes": [ + 10000000 + ], + "vector_dim": 128, + "queries_per_size": 2000, + "top_k": 10, + "seed": 42, + "noise": 0.08, + "batch_size": 10000, + "target_recall_definition": "source id appears in top-k for a noisy copy of that source vector", + "memory_model": "streaming batches; stores only selected query source vectors outside the index", + "default_target_sizes": [ + 10000000, + 50000000 + ], + "slo_targets": { + "target_recall_at_k": 0.95, + "target_p99_ms": 100.0, + "target_qps": 250.0, + "replicas": 4, + "autoscaling_max_replicas": 48, + "capacity_headroom": 0.7 + }, + "cost_model": { + "replica_hourly_cost_usd": 0.25, + "storage_gb_monthly_cost_usd": 0.1, + "memory_payload_kb": 2.0, + "vector_dtype_bytes": 4, + "hours_per_month": 730.0, + "monthly_budget_usd": null, + "max_cost_per_1m_memories_usd": null, + "max_compute_cost_per_1m_queries_usd": null + } + }, + "preflight": { + "python": "3.13.3", + "platform": "Windows-11-10.0.26200-SP0", + "docker": { + "available": true, + "server_version": "28.3.0" + }, + "modules": { + "faiss": false, + "qdrant_client": true, + "psycopg": false, + "numpy": true + }, + "environment": { + "WAVEMIND_FAISS_PATH": false, + "WAVEMIND_QDRANT_URL": false, + "WAVEMIND_PGVECTOR_DSN": false, + "WAVEMIND_PGVECTOR_CREATE_HNSW": null, + "WAVEMIND_PGVECTOR_EF_SEARCH": null, + "WAVEMIND_QDRANT_HNSW_EF": "1024", + "WAVEMIND_QDRANT_EXACT": null, + "WAVEMIND_QDRANT_WARMUP_QUERIES": "100", + "WAVEMIND_QDRANT_WAIT_AFTER_BUILD_SECONDS": "10", + "WAVEMIND_QDRANT_INDEX_READY_TIMEOUT_SECONDS": "21600", + "WAVEMIND_QDRANT_INDEX_READY_POLL_SECONDS": "10", + "WAVEMIND_QDRANT_REQUIRE_FULL_INDEX": "1", + "WAVEMIND_QDRANT_DEFER_INDEXING": "1", + "WAVEMIND_QDRANT_DEFERRED_INDEXING_THRESHOLD_KB": "1000000000", + "WAVEMIND_QDRANT_FINAL_INDEXING_THRESHOLD_KB": "20000", + "WAVEMIND_QDRANT_HNSW_M": "16", + "WAVEMIND_QDRANT_HNSW_EF_CONSTRUCT": "100", + "WAVEMIND_QDRANT_HNSW_FULL_SCAN_THRESHOLD": null, + "WAVEMIND_QDRANT_OPTIMIZER_DEFAULT_SEGMENT_NUMBER": "4", + "WAVEMIND_QDRANT_OPTIMIZER_INDEXING_THRESHOLD": null, + "WAVEMIND_QDRANT_VECTOR_ON_DISK": "1", + "WAVEMIND_QDRANT_ON_DISK_PAYLOAD": "1", + "WAVEMIND_QDRANT_SHARD_NUMBER": null + }, + "disk": { + "root": "D:\\", + "free_gb": 2.51, + "total_gb": 931.51 + } + }, + "results": [ + { + "vectors": 10000000, + "vector_dim": 128, + "queries": 2000, + "top_k": 10, + "noise": 0.08, + "batch_size": 10000, + "results": [ + { + "engine": "Qdrant sharded service streaming", + "vectors": 10000000, + "vector_dim": 128, + "batch_size": 10000, + "recall_at_k": 0.9925, + "target_recall_at_k": 0.9925, + "target_recall_at_1": 0.9925, + "avg_latency_ms": 46.60563879973779, + "p50_latency_ms": 45.0464000023203, + "p95_latency_ms": 60.49189998884685, + "p99_latency_ms": 71.27849999233149, + "max_latency_ms": 125.04669997724704, + "build_ms": 5520602.228099975, + "queries": 2000, + "recall_definition": "source vector id appears in top_k for a noisy copy of that vector", + "collection_prefix": "wavemind_qdrant_sharded_10m_v1182", + "collection_names": [ + "wavemind_qdrant_sharded_10m_v1182_s000", + "wavemind_qdrant_sharded_10m_v1182_s001", + "wavemind_qdrant_sharded_10m_v1182_s002", + "wavemind_qdrant_sharded_10m_v1182_s003" + ], + "shard_count": 4, + "fanout_workers": 4, + "parallel_shard_upsert": true, + "routing": "point_id_minus_one_mod_shard_count", + "warmup_queries": 100, + "wait_after_build_seconds": 10.0, + "index_readiness": [ + { + "collection_name": "wavemind_qdrant_sharded_10m_v1182_s000", + "expected_vectors": 2500000, + "points_count": 2500000, + "indexed_vectors_count": 2500000, + "collection_status": "green", + "optimizer_status": "ok", + "ready": true, + "attempts": 235, + "wait_ms": 2347914.449499978, + "timeout_seconds": 21600.0, + "poll_interval_seconds": 10.0, + "require_full_index": true + }, + { + "collection_name": "wavemind_qdrant_sharded_10m_v1182_s001", + "expected_vectors": 2500000, + "points_count": 2500000, + "indexed_vectors_count": 2500000, + "collection_status": "green", + "optimizer_status": "ok", + "ready": true, + "attempts": 78, + "wait_ms": 772670.2865000116, + "timeout_seconds": 21600.0, + "poll_interval_seconds": 10.0, + "require_full_index": true + }, + { + "collection_name": "wavemind_qdrant_sharded_10m_v1182_s002", + "expected_vectors": 2500000, + "points_count": 2500000, + "indexed_vectors_count": 2500000, + "collection_status": "green", + "optimizer_status": "ok", + "ready": true, + "attempts": 1, + "wait_ms": 120.68679998628795, + "timeout_seconds": 21600.0, + "poll_interval_seconds": 10.0, + "require_full_index": true + }, + { + "collection_name": "wavemind_qdrant_sharded_10m_v1182_s003", + "expected_vectors": 2500000, + "points_count": 2500000, + "indexed_vectors_count": 2500000, + "collection_status": "green", + "optimizer_status": "ok", + "ready": true, + "attempts": 1, + "wait_ms": 43.11050000251271, + "timeout_seconds": 21600.0, + "poll_interval_seconds": 10.0, + "require_full_index": true + } + ], + "index_ready_all": true, + "deferred_indexing": { + "enabled": true, + "deferred_threshold_kb": 1000000000, + "final_threshold_kb": 20000 + }, + "index_restore_ms": 1727.1943000087049, + "search_params": { + "hnsw_ef": 1024, + "exact": false, + "quantization": { + "enabled": true, + "ignore": null, + "rescore": false, + "oversampling": null + } + }, + "transport": { + "prefer_grpc": true, + "grpc_ports": [ + 6352, + 6354, + 6356, + 6358 + ], + "query_timeout_seconds": 60 + }, + "collection_params": { + "hnsw": { + "m": 16, + "ef_construct": 100, + "max_indexing_threads": 2, + "on_disk": false + }, + "optimizers": { + "default_segment_number": 4 + }, + "vector_on_disk": true, + "on_disk_payload": true, + "shard_number": null, + "scalar_quantization": { + "type": "int8", + "quantile": 0.99, + "always_ram": true + } + }, + "upsert_batch_size": 5000, + "qdrant_checkpoint_complete_resume": false, + "memory_mode": "horizontally sharded streaming upsert; parallel fanout query merge", + "checkpoint_enabled": true, + "checkpoint_path": "qdrant-sharded-service-10000000.checkpoint.json", + "checkpoint_completed_batches": 1000, + "checkpoint_source_vectors": 2000, + "slo_status": "scale_required", + "slo_required_replicas": 17, + "slo_autoscaled_qps": 720.9428057488408, + "cost_status": "valid_slo", + "compute_cost_per_1m_queries_usd": 4.722222222222222, + "monthly_storage_cost_usd": 2.384185791015625, + "monthly_total_cost_at_target_qps_usd": 3104.8841857910156, + "estimated_storage_gb": 23.84185791015625 + } + ], + "slo": [ + { + "engine": "Qdrant sharded service streaming", + "status": "scale_required", + "target_recall_at_k": 0.95, + "target_p99_ms": 100.0, + "target_qps": 250.0, + "recall_at_k": 0.9925, + "p99_latency_ms": 71.27849999233149, + "avg_latency_ms": 46.60563879973779, + "per_replica_qps_at_headroom": 15.019641786434182, + "current_replicas": 4, + "current_capacity_qps": 60.07856714573673, + "required_replicas": 17, + "autoscaling_max_replicas": 48, + "autoscaled_capacity_qps": 720.9428057488408, + "blocking_reasons": [], + "cost_status": "valid_slo", + "compute_cost_per_1m_queries_usd": 4.722222222222222, + "monthly_storage_cost_usd": 2.384185791015625, + "monthly_total_cost_at_target_qps_usd": 3104.8841857910156, + "estimated_storage_gb": 23.84185791015625 + } + ], + "cost": [ + { + "engine": "Qdrant sharded service streaming", + "cost_status": "valid_slo", + "cost_blocking_reasons": [], + "memory_count": 10000000, + "vector_dim": 128, + "required_replicas": 17, + "target_qps": 250.0, + "replica_hourly_cost_usd": 0.25, + "storage_gb_monthly_cost_usd": 0.1, + "monthly_budget_usd": null, + "monthly_budget_headroom_usd": null, + "max_cost_per_1m_memories_usd": null, + "max_compute_cost_per_1m_queries_usd": null, + "vector_storage_gb": 4.76837158203125, + "payload_storage_gb": 19.073486328125, + "total_storage_gb": 23.84185791015625, + "monthly_storage_cost_usd": 2.384185791015625, + "monthly_storage_cost_per_1m_memories_usd": 0.2384185791015625, + "monthly_compute_cost_per_1m_memories_usd": 310.25, + "monthly_total_cost_per_1m_memories_usd": 310.48841857910156, + "monthly_queries_at_target_qps": 657000000.0, + "compute_cost_per_1m_queries_usd": 4.722222222222222, + "monthly_compute_cost_at_target_qps_usd": 3102.5, + "monthly_total_cost_at_target_qps_usd": 3104.8841857910156 + } + ] + } + ] +} diff --git a/benchmarks/release_claims_results.json b/benchmarks/release_claims_results.json index e0bf636..81aecfa 100644 --- a/benchmarks/release_claims_results.json +++ b/benchmarks/release_claims_results.json @@ -1,6 +1,6 @@ { "schema": "wavemind.release_claims.v1", - "generated_at": "2026-07-11T03:38:03Z", + "generated_at": "2026-07-11T05:56:52Z", "release_status": "core_release_ready", "claim_status": "claims_limited", "summary": { @@ -11,7 +11,7 @@ "artifact_audit_status": "pass", "allowed_claim_count": 3, "locked_claim_count": 2, - "next_action_count": 5 + "next_action_count": 4 }, "allowed_claims": [ { @@ -68,18 +68,6 @@ "command": "gh workflow run serverless-observed-telemetry.yml -f nodes=\"$WAVEMIND_SERVERLESS_NODES\" -f seed_mode=first -f commit_results=true", "claim_unlocked": "Hosted/serverless p99, cold-start, error-rate, and scale-out SLO." }, - { - "id": "qdrant_sharded_10m_service", - "title": "10M sharded Qdrant service load", - "strict_status": "action_required", - "preflight_status": "action_required", - "artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", - "missing_env": [ - "WAVEMIND_QDRANT_URLS" - ], - "command": "python benchmarks/production_streaming_load_benchmark.py --sizes 10000000 --dim 128 --queries 2000 --top-k 10 --seed 42 --noise 0.08 --batch-size 5000 --engines qdrant-sharded-service --target-recall 0.95 --target-p99-ms 100.0 --target-qps 250.0 --replicas 4 --autoscaling-max-replicas 48 --capacity-headroom 0.7 --output benchmarks/production_streaming_load_qdrant_sharded_10m_results.json --checkpoint-path state/production-runs/qdrant-sharded-service-10000000.checkpoint.json", - "claim_unlocked": "Horizontally sharded Qdrant service recall/latency SLO." - }, { "id": "pgvector_10m_service", "title": "10M pgvector service load", diff --git a/benchmarks/render_benchmark_leaderboard.py b/benchmarks/render_benchmark_leaderboard.py index ad379df..1fb358d 100644 --- a/benchmarks/render_benchmark_leaderboard.py +++ b/benchmarks/render_benchmark_leaderboard.py @@ -496,6 +496,17 @@ def evidence_status_rows(payload: dict[str, Any], root: Path = PROJECT_ROOT) -> nested_results = qdrant_sharded_smoke_result["results"] if nested_results: qdrant_sharded_smoke_result = nested_results[0] + qdrant_sharded_10m = load_json_if_exists( + root, + "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", + ) + qdrant_sharded_10m_result = _first_result(qdrant_sharded_10m) + if qdrant_sharded_10m_result and isinstance( + qdrant_sharded_10m_result.get("results"), list + ): + nested_results = qdrant_sharded_10m_result["results"] + if nested_results: + qdrant_sharded_10m_result = nested_results[0] qdrant_sharded_plan = load_json_if_exists(root, "benchmarks/production_streaming_load_qdrant_sharded_10m_plan.json") qdrant_sharded_100m_plan = load_json_if_exists(root, "benchmarks/production_streaming_load_qdrant_sharded_100m_plan.json") qdrant_sharded_plan_row = {} @@ -519,20 +530,39 @@ def evidence_status_rows(payload: dict[str, Any], root: Path = PROJECT_ROOT) -> if qdrant_sharded_smoke_result and qdrant_sharded_plan_row: shard_urls = qdrant_sharded_plan_row.get("command_env", {}).get("WAVEMIND_QDRANT_URLS", "") shard_count = len([part for part in str(shard_urls).split(",") if part.strip()]) - blockers = ", ".join(qdrant_sharded_plan_row.get("blockers", [])) or "none" + blockers = ( + "none (measured artifact passes)" + if qdrant_sharded_10m_result + else ", ".join(qdrant_sharded_plan_row.get("blockers", [])) or "none" + ) hundred_million_status = qdrant_sharded_100m_plan_row.get("status", "missing") + measured_readout = ( + f"10M recall `{fmt(qdrant_sharded_10m_result.get('target_recall_at_k'))}`, " + f"10M p99 `{fmt(qdrant_sharded_10m_result.get('p99_latency_ms'))} ms`, " + f"shards `{qdrant_sharded_10m_result.get('shard_count', '?')}`" + if qdrant_sharded_10m_result + else f"10M preflight `{qdrant_sharded_plan_row.get('status', 'unknown')}`" + ) rows.append( ( "Qdrant sharded streaming", - "real two-service fanout smoke plus horizontal Qdrant preflight", + ( + "real fanout smoke plus measured four-service 10M profile" + if qdrant_sharded_10m_result + else "real two-service fanout smoke plus horizontal Qdrant preflight" + ), ( f"smoke recall `{fmt(qdrant_sharded_smoke_result.get('target_recall_at_k'))}`, " f"smoke p99 `{fmt(qdrant_sharded_smoke_result.get('p99_latency_ms'))} ms`; " - f"10M preflight `{qdrant_sharded_plan_row.get('status', 'unknown')}`; " + f"{measured_readout}; " f"100M preflight `{hundred_million_status}`; " f"planned shards `{shard_count}`; blockers `{blockers}`" ), - "Run `.github/workflows/production-streaming-load.yml` with `qdrant-sharded-service` and publish `benchmarks/production_streaming_load_qdrant_sharded_10m_results.json` or `benchmarks/production_streaming_load_qdrant_sharded_100m_results.json`.", + ( + "Keep the measured 10M sharded profile green and run the strict 100M sharded profile next." + if qdrant_sharded_10m_result + else "Run `.github/workflows/production-streaming-load.yml` with `qdrant-sharded-service` and publish the 10M or 100M result artifact." + ), ) ) diff --git a/benchmarks/scale_gap_results.json b/benchmarks/scale_gap_results.json index fd4ff03..3c52f95 100644 --- a/benchmarks/scale_gap_results.json +++ b/benchmarks/scale_gap_results.json @@ -1,16 +1,16 @@ { "schema": "wavemind.scale_gap.v1", - "generated_at": "2026-07-11T03:38:04Z", + "generated_at": "2026-07-11T05:56:53Z", "overall_status": "action_required", "summary": { "total_profiles": 5, - "complete_count": 2, + "complete_count": 3, "ready_to_run_count": 0, - "blocked_by_env_count": 3, + "blocked_by_env_count": 2, "blocked_by_preflight_count": 0, "missing_plan_count": 0, "planned_target_memories": 180000000, - "proven_target_memories": 60000000, + "proven_target_memories": 70000000, "nearest_baseline_max_memories": 10000000, "claim_status": "claims_limited" }, @@ -57,8 +57,8 @@ { "profile": "qdrant-sharded-10m", "requirement_id": "qdrant_sharded_10m_service", - "status": "blocked_by_env", - "strict_status": "action_required", + "status": "complete", + "strict_status": "pass", "plan_status": "action_required", "preflight_status": "action_required", "engine": "qdrant-sharded-service", @@ -67,7 +67,7 @@ "target_p99_ms": 100.0, "target_qps": 250.0, "output_artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", - "output_artifact_exists": false, + "output_artifact_exists": true, "checkpoint_path": "state/production-runs/qdrant-sharded-service-10000000.checkpoint.json", "missing_env": [ "WAVEMIND_QDRANT_URLS" @@ -91,7 +91,7 @@ }, "baseline_progress_ratio": 0.1, "target_gap_multiplier": 10.0, - "next_action": "Provision the listed environment, run the command, then promote the result artifact through the ingest gate." + "next_action": "Strict result artifact already passes." }, { "profile": "pgvector-10m", diff --git a/benchmarks/strict_evidence_readiness_results.json b/benchmarks/strict_evidence_readiness_results.json index 5548fc7..56459b4 100644 --- a/benchmarks/strict_evidence_readiness_results.json +++ b/benchmarks/strict_evidence_readiness_results.json @@ -1,6 +1,6 @@ { "schema": "wavemind.strict_evidence_readiness.v1", - "generated_at": "2026-07-11T03:38:04Z", + "generated_at": "2026-07-11T05:56:54Z", "status": "pass", "readiness_status": "action_required", "claim_status": "claims_limited", @@ -9,8 +9,8 @@ "readiness_status": "action_required", "claim_status": "claims_limited", "total_requirements": 8, - "action_required_count": 5, - "complete_count": 3, + "action_required_count": 4, + "complete_count": 4, "ready_for_safe_dispatch_count": 0, "can_auto_run_now_count": 0, "target_memories_total": 180000000, @@ -18,12 +18,12 @@ "pass": 8 }, "dispatch_status_counts": { - "blocked_by_preflight": 5, - "complete": 3 + "blocked_by_preflight": 4, + "complete": 4 }, "blocker_counts": { - "complete": 3, - "missing_env": 5 + "complete": 4, + "missing_env": 4 } }, "checks": [ @@ -257,14 +257,14 @@ { "id": "qdrant_sharded_10m_service", "title": "10M sharded Qdrant service load", - "strict_status": "action_required", + "strict_status": "pass", "preflight_status": "action_required", - "dispatch_status": "blocked_by_preflight", - "scale_gap_status": "blocked_by_env", + "dispatch_status": "complete", + "scale_gap_status": "complete", "workflow": "production-streaming-load.yml", "wave": "service-scale-10m", "artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", - "artifact_exists": false, + "artifact_exists": true, "output_artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", "claim_unlocked": "Horizontally sharded Qdrant service recall/latency SLO.", "locked_claim": "10M-100M service-backed production scale", @@ -294,8 +294,8 @@ "post_ingest_refresh_command": "python benchmarks/strict_evidence_readiness_report.py --output benchmarks/strict_evidence_readiness_results.json --markdown-output benchmarks/STRICT_EVIDENCE_READINESS.md", "ready_for_safe_dispatch": false, "can_auto_run_now": false, - "next_action": "Provision the listed environment, run the command, then promote the result artifact through the ingest gate.", - "blocker_category": "missing_env" + "next_action": "Strict result artifact already passes.", + "blocker_category": "complete" }, { "id": "pgvector_10m_service", diff --git a/benchmarks/structured_memory_results.json b/benchmarks/structured_memory_results.json index c754554..b7eadb9 100644 --- a/benchmarks/structured_memory_results.json +++ b/benchmarks/structured_memory_results.json @@ -1,7 +1,7 @@ { "schema": "wavemind.structured_memory_report.v1", "generated_at": "2026-07-09T22:45:10Z", - "source_ref": "be2cebfd776b", + "source_ref": "c4f786e131c8", "source_file": "benchmarks/scale_readiness_results.json", "claim_boundary": "Structured-memory rows come from the checked-in scale-readiness artifact. They prove typed payload routing, provenance, persistence, temporal recall, and graph traversal on the deterministic fixture; they do not claim full production multimodal model quality.", "summary": { diff --git a/docs/benchmark-dashboard.html b/docs/benchmark-dashboard.html index 33a3f5c..d05de81 100644 --- a/docs/benchmark-dashboard.html +++ b/docs/benchmark-dashboard.html @@ -70,7 +70,7 @@

WaveMind Living Benchmark Dashboard

Readiness

pass

39/39 criteria pass

Implemented

37

3 runner-ready and 6 planned public proof paths

-

Refresh

2026-07-11T03:37:56Z

source be2cebfd776b

+

Refresh

2026-07-11T05:56:44Z

source c4f786e131c8

@@ -78,7 +78,7 @@

Visual Summary

WaveMind benchmark summary
-

Publication Contract

The leaderboard is generated from artifacts, freshness-checked, published to GitHub Pages, and claim-limited until strict production evidence passes.

Statuspass
Weekly schedule17 4 * * 1
Refresh profilelocal
Pages URLhttps://caspiang.github.io/wavemind/
Source refbe2cebfd776b
Workflow runlocal or manual artifact
weekly schedule: truemanual dispatch: truegithub pages upload: truegithub pages deploy: truereview artifact uploaded: trueno scheduled bot commit to main: truestrict freshness gate: truemachine status published: true
+

Publication Contract

The leaderboard is generated from artifacts, freshness-checked, published to GitHub Pages, and claim-limited until strict production evidence passes.

Statuspass
Weekly schedule17 4 * * 1
Refresh profilelocal
Pages URLhttps://caspiang.github.io/wavemind/
Source refc4f786e131c8
Workflow runlocal or manual artifact
weekly schedule: truemanual dispatch: truegithub pages upload: truegithub pages deploy: truereview artifact uploaded: trueno scheduled bot commit to main: truestrict freshness gate: truemachine status published: true

Agent Impact

Behavioral evidence: task success, stale-fact suppression, context savings, long-memory retrieval, and checked-in answer-quality smoke results.

Statuspass
Benchmarks6
WaveMind wins6
Average lift0.37
Context saved0.719
Stale safety1
Best profileagent-coherence-and-token-savings-wavemind

Read the agent impact report

@@ -90,7 +90,7 @@

Visual Summary

Cluster Autoscale

Cluster evidence: shard placement, autoscale planning, Kubernetes operator reconciliation, rebalance safety, active-active convergence, CRDT field state, and the deterministic 100M capacity envelope.

Statuspass
Gate checks62/62
Simulated memories1000000
Namespaces4096
Autoscale target10000000
Required nodes50
Operator replicas34
100M capacity nodes128
100M capacity zones8
Recommended max replicas192

Read the cluster autoscale report

-

Strict Evidence Readiness

Operator runbook for the remaining remote, 10M, 50M, and 100M evidence gaps: safe dispatch commands, missing environment, promotion steps, strict validation, and locked claims.

Blockers: complete: 3, missing_env: 5

Report statuspass
Readinessaction_required
Claim statusclaims_limited
Requirements8
Action required5
Safe dispatch ready0
Can auto-run now0
Planned target memories180000000

Read the strict evidence readiness runbook

+

Strict Evidence Readiness

Operator runbook for the remaining remote, 10M, 50M, and 100M evidence gaps: safe dispatch commands, missing environment, promotion steps, strict validation, and locked claims.

Blockers: complete: 4, missing_env: 4

Report statuspass
Readinessaction_required
Claim statusclaims_limited
Requirements8
Action required4
Safe dispatch ready0
Can auto-run now0
Planned target memories180000000

Read the strict evidence readiness runbook

Benchmark Leaderboard

@@ -293,8 +293,8 @@

Evidence Source Status

Artifact freshness -local matrix refresh at 2026-07-11T03:37:56Z -source be2cebfd776b; audit gate enforced by validate_benchmark_artifacts.py +local matrix refresh at 2026-07-11T05:56:44Z +source c4f786e131c8; audit gate enforced by validate_benchmark_artifacts.py Keep weekly refresh green before public claims. @@ -347,9 +347,9 @@

Evidence Source Status

Qdrant sharded streaming -real two-service fanout smoke plus horizontal Qdrant preflight -smoke recall 1, smoke p99 16.0 ms; 10M preflight action_required; 100M preflight action_required; planned shards 4; blockers missing_env:WAVEMIND_QDRANT_URLS, insufficient_local_disk_for_index_and_transient_batches -Run .github/workflows/production-streaming-load.yml with qdrant-sharded-service and publish benchmarks/production_streaming_load_qdrant_sharded_10m_results.json or benchmarks/production_streaming_load_qdrant_sharded_100m_results.json. +real fanout smoke plus measured four-service 10M profile +smoke recall 1, smoke p99 16.0 ms; 10M recall 0.993, 10M p99 71.3 ms, shards 4; 100M preflight action_required; planned shards 4; blockers none (measured artifact passes) +Keep the measured 10M sharded profile green and run the strict 100M sharded profile next. Qdrant 1M streaming diff --git a/docs/data/leaderboard-status.json b/docs/data/leaderboard-status.json index 796fae5..acee7a4 100644 --- a/docs/data/leaderboard-status.json +++ b/docs/data/leaderboard-status.json @@ -1,7 +1,7 @@ { "schema": "wavemind.leaderboard_status.v1", - "generated_at": "2026-07-11T03:38:05Z", - "source_ref": "be2cebfd776b", + "generated_at": "2026-07-11T05:56:54Z", + "source_ref": "c4f786e131c8", "workflow_run_id": null, "refresh_profile": "local", "public_url": "https://caspiang.github.io/wavemind/", @@ -14,7 +14,7 @@ "timezone": "UTC", "public_url": "https://caspiang.github.io/wavemind/", "publishing_status": "publishable_with_claim_limits", - "source_ref": "be2cebfd776b", + "source_ref": "c4f786e131c8", "workflow_run_id": null, "refresh_profile": "local", "expected_scheduled_refresh_profile": "weekly-fast", @@ -39,7 +39,7 @@ "freshness_gate": { "schema": "wavemind.leaderboard_freshness.v1", "status": "pass", - "checked_at": "2026-07-11T03:38:05Z", + "checked_at": "2026-07-11T05:56:54Z", "max_age_days": 8.0, "source_count": 32, "fresh_count": 32, @@ -55,15 +55,15 @@ "path": "benchmarks/benchmark_matrix_results.json", "schema": "wavemind.benchmark_matrix.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:37:56Z", - "age_days": 0.00010416666666666667, + "timestamp": "2026-07-11T05:56:44Z", + "age_days": 0.00011574074074074075, "status": "pass" }, { "path": "benchmarks/benchmark_artifact_audit.json", "schema": "wavemind.benchmark_artifact_audit.v1", "timestamp_key": "checked_at", - "timestamp": "2026-07-11T03:38:05Z", + "timestamp": "2026-07-11T05:56:54Z", "age_days": 0.0, "status": "pass" }, @@ -71,7 +71,7 @@ "path": "benchmarks/production_readiness_results.json", "schema": "wavemind.production_readiness.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:37:59Z", + "timestamp": "2026-07-11T05:56:48Z", "age_days": 6.944444444444444e-05, "status": "pass" }, @@ -79,7 +79,7 @@ "path": "benchmarks/production_evidence_results.json", "schema": "wavemind.production_evidence.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:38:00Z", + "timestamp": "2026-07-11T05:56:49Z", "age_days": 5.787037037037037e-05, "status": "pass" }, @@ -87,7 +87,7 @@ "path": "benchmarks/production_evidence_preflight_results.json", "schema": "wavemind.production_evidence_preflight.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:38:01Z", + "timestamp": "2026-07-11T05:56:50Z", "age_days": 4.6296296296296294e-05, "status": "pass" }, @@ -96,14 +96,14 @@ "schema": "wavemind.production_evidence_env_contract.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-10T05:23:37Z", - "age_days": 0.926712962962963, + "age_days": 1.023113425925926, "status": "pass" }, { "path": "benchmarks/production_evidence_bundle_results.json", "schema": "wavemind.production_evidence_bundle.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:38:02Z", + "timestamp": "2026-07-11T05:56:51Z", "age_days": 3.472222222222222e-05, "status": "pass" }, @@ -111,7 +111,7 @@ "path": "benchmarks/release_claims_results.json", "schema": "wavemind.release_claims.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:38:03Z", + "timestamp": "2026-07-11T05:56:52Z", "age_days": 2.3148148148148147e-05, "status": "pass" }, @@ -119,7 +119,7 @@ "path": "benchmarks/scale_gap_results.json", "schema": "wavemind.scale_gap.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:38:04Z", + "timestamp": "2026-07-11T05:56:53Z", "age_days": 1.1574074074074073e-05, "status": "pass" }, @@ -127,8 +127,8 @@ "path": "benchmarks/strict_evidence_readiness_results.json", "schema": "wavemind.strict_evidence_readiness.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:38:04Z", - "age_days": 1.1574074074074073e-05, + "timestamp": "2026-07-11T05:56:54Z", + "age_days": 0.0, "status": "pass" }, { @@ -136,7 +136,7 @@ "schema": "wavemind.cluster_admission.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-10T05:23:44Z", - "age_days": 0.9266319444444444, + "age_days": 1.0230324074074073, "status": "pass" }, { @@ -144,7 +144,7 @@ "schema": "wavemind.active_active_admission.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T10:20:56Z", - "age_days": 1.7202430555555555, + "age_days": 1.8166435185185186, "status": "pass" }, { @@ -152,7 +152,7 @@ "schema": "wavemind.serverless_admission.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T10:20:56Z", - "age_days": 1.7202430555555555, + "age_days": 1.8166435185185186, "status": "pass" }, { @@ -160,7 +160,7 @@ "schema": "wavemind.multimodal_admission.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T12:56:09Z", - "age_days": 1.6124537037037037, + "age_days": 1.7088541666666666, "status": "pass" }, { @@ -168,7 +168,7 @@ "schema": "wavemind.memory_os_admission.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T00:40:28Z", - "age_days": 2.1233449074074073, + "age_days": 2.2197453703703705, "status": "pass" }, { @@ -176,7 +176,7 @@ "schema": "wavemind.memory_os_canary.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T01:06:28Z", - "age_days": 2.105289351851852, + "age_days": 2.2016898148148147, "status": "pass" }, { @@ -184,7 +184,7 @@ "schema": "wavemind.memory_os_policy_evolution.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T13:44:41Z", - "age_days": 1.57875, + "age_days": 1.675150462962963, "status": "pass" }, { @@ -192,14 +192,14 @@ "schema": "wavemind.memory_os_policy_bundle.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T19:29:13Z", - "age_days": 1.3394907407407408, + "age_days": 1.4358912037037037, "status": "pass" }, { "path": "benchmarks/production_evidence_dispatch_results.json", "schema": "wavemind.production_evidence_dispatch.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:38:02Z", + "timestamp": "2026-07-11T05:56:51Z", "age_days": 3.472222222222222e-05, "status": "pass" }, @@ -208,7 +208,7 @@ "schema": "wavemind.production_scale_run_plan.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-10T04:17:18Z", - "age_days": 0.9727662037037037, + "age_days": 1.0691666666666666, "status": "pass" }, { @@ -216,15 +216,15 @@ "schema": "wavemind.agent_coherence_benchmark.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-08T17:24:13Z", - "age_days": 2.426296296296296, + "age_days": 2.5226967592592593, "status": "pass" }, { "path": "benchmarks/agent_impact_results.json", "schema": "wavemind.agent_impact_leaderboard.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:37:56Z", - "age_days": 0.00010416666666666667, + "timestamp": "2026-07-11T05:56:44Z", + "age_days": 0.00011574074074074075, "status": "pass" }, { @@ -232,7 +232,7 @@ "schema": "wavemind.structured_memory_report.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T22:45:10Z", - "age_days": 1.2034143518518519, + "age_days": 1.2998148148148148, "status": "pass" }, { @@ -240,7 +240,7 @@ "schema": "wavemind.memory_os_intelligence_report.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T22:45:10Z", - "age_days": 1.2034143518518519, + "age_days": 1.2998148148148148, "status": "pass" }, { @@ -248,7 +248,7 @@ "schema": "wavemind.cluster_autoscale_report.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T22:45:10Z", - "age_days": 1.2034143518518519, + "age_days": 1.2998148148148148, "status": "pass" }, { @@ -256,7 +256,7 @@ "schema": "wavemind.kubernetes_operator_smoke.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T23:28:34.759347Z", - "age_days": 1.173266674224537, + "age_days": 1.2696671371875, "status": "pass" }, { @@ -264,7 +264,7 @@ "schema": "wavemind.kubernetes_cluster_network_smoke.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-10T05:13:44.475824+00:00", - "age_days": 0.9335708816666667, + "age_days": 1.0299713446296297, "status": "pass" }, { @@ -272,7 +272,7 @@ "schema": "wavemind.kubernetes_active_active_region_smoke.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T23:55:54.133679+00:00", - "age_days": 1.1542924342708332, + "age_days": 1.2506928972337963, "status": "pass" }, { @@ -280,7 +280,7 @@ "schema": "wavemind.kubernetes_serverless_lifecycle_smoke.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-10T02:42:30.318307+00:00", - "age_days": 1.0385958529282409, + "age_days": 1.1349963158912038, "status": "pass" }, { @@ -288,7 +288,7 @@ "schema": "wavemind.kubernetes_postgres_qdrant_dr_smoke.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-10T02:42:53.084317+00:00", - "age_days": 1.0383323574421297, + "age_days": 1.1347328204050926, "status": "pass" }, { @@ -296,22 +296,22 @@ "schema": "wavemind.scale_readiness_benchmark.v1", "timestamp_key": "generated_at", "timestamp": "2026-07-09T22:45:10Z", - "age_days": 1.2034143518518519, + "age_days": 1.2998148148148148, "status": "pass" }, { "path": "benchmarks/cost_efficiency_results.json", "schema": "wavemind.cost_efficiency_leaderboard.v1", "timestamp_key": "generated_at", - "timestamp": "2026-07-11T03:37:56Z", - "age_days": 0.00010416666666666667, + "timestamp": "2026-07-11T05:56:44Z", + "age_days": 0.00011574074074074075, "status": "pass" } ] }, "benchmark_matrix": { "schema": "wavemind.benchmark_matrix.v1", - "generated_at": "2026-07-11T03:37:56Z", + "generated_at": "2026-07-11T05:56:44Z", "implemented_count": 37, "runner_ready_count": 3, "planned_count": 6, @@ -340,8 +340,8 @@ "artifact_audit": { "schema": "wavemind.benchmark_artifact_audit.v1", "status": "pass", - "checked_at": "2026-07-11T03:38:05Z", - "age_days": 0.00010598428240740742, + "checked_at": "2026-07-11T05:56:54Z", + "age_days": 0.00012261520833333332, "max_age_days": 8.0, "errors": [] }, @@ -886,8 +886,8 @@ "overall_status": "action_required", "summary": { "overall_status": "action_required", - "pass_count": 3, - "action_required_count": 5, + "pass_count": 4, + "action_required_count": 4, "fail_count": 0, "total_requirements": 8 }, @@ -910,15 +910,6 @@ "missing artifact" ] }, - { - "id": "qdrant_sharded_10m_service", - "title": "10M sharded Qdrant service load", - "artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", - "claim_unlocked": "Horizontally sharded Qdrant service recall/latency SLO.", - "issues": [ - "missing artifact" - ] - }, { "id": "pgvector_10m_service", "title": "10M pgvector service load", @@ -945,7 +936,7 @@ "summary": { "claim_status": "claims_limited", "strict_overall_status": "action_required", - "strict_pass_count": 3, + "strict_pass_count": 4, "strict_total_requirements": 8, "preflight_overall_status": "action_required", "preflight_ready_count": 0, @@ -957,9 +948,9 @@ "production_scale_run_contract_status": "available", "production_scale_run_profile_count": 5, "production_scale_run_target_memories_total": 180000000, - "next_action_count": 5 + "next_action_count": 4 }, - "next_action_count": 5, + "next_action_count": 4, "production_scale_run_contract": { "status": "available", "artifact": "benchmarks/production_scale_run_plan.json", @@ -1089,10 +1080,10 @@ }, "cost_efficiency": { "schema": "wavemind.cost_efficiency_leaderboard.v1", - "measured_row_count": 21, + "measured_row_count": 22, "planned_row_count": 5, "measured_slo_pass_count": 9, - "measured_valid_cost_count": 12, + "measured_valid_cost_count": 13, "planned_valid_cost_count": 5, "measured_frontier_profiles": [ "sub_100k-wavemind-numpy-streaming-production_streaming_load_smoke_results", @@ -1100,6 +1091,7 @@ "1m-qdrant-service-streaming-production_streaming_load_qdrant_1m_tuned_results", "1m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_1m_results", "100k-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_100k_results", + "10m-qdrant-sharded-service-streaming-production_streaming_load_qdrant_sharded_10m_results", "10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results", "10m-qdrant-service-streaming-production_streaming_load_qdrant_10m_results" ], @@ -1109,7 +1101,7 @@ ], "best_measured_by_target_class": { "100k": "100k-qdrant-service-production_load_qdrant_100k_tuned_results", - "10m": "10m-wavemind-faiss-ivfpq-persisted-streaming-production_streaming_load_ivfpq_10m_results", + "10m": "10m-qdrant-sharded-service-streaming-production_streaming_load_qdrant_sharded_10m_results", "1m": "1m-qdrant-service-streaming-production_streaming_load_qdrant_1m_tuned_results", "sub_100k": "sub_100k-wavemind-numpy-streaming-production_streaming_load_smoke_results" }, @@ -1172,8 +1164,8 @@ "overall_status": "action_required", "total_jobs": 8, "ready_to_dispatch_count": 0, - "blocked_by_preflight_count": 5, - "complete_count": 3, + "blocked_by_preflight_count": 4, + "complete_count": 4, "commit_results_default": false, "runner_label": "self-hosted-large", "wave_counts": { @@ -1183,8 +1175,8 @@ "service-scale-10m": 3 }, "status_counts": { - "blocked_by_preflight": 5, - "complete": 3 + "blocked_by_preflight": 4, + "complete": 4 } }, "jobs": [ @@ -1389,10 +1381,10 @@ { "id": "qdrant_sharded_10m_service", "title": "10M sharded Qdrant service load", - "status": "blocked_by_preflight", - "dispatch_required": true, + "status": "complete", + "dispatch_required": false, "ready": false, - "strict_status": "action_required", + "strict_status": "pass", "preflight_status": "action_required", "wave": "service-scale-10m", "workflow": "production-streaming-load.yml", @@ -1600,7 +1592,7 @@ "artifact_audit_status": "pass", "allowed_claim_count": 3, "locked_claim_count": 2, - "next_action_count": 5 + "next_action_count": 4 }, "allowed_claims": [ { @@ -1637,13 +1629,13 @@ "overall_status": "action_required", "summary": { "total_profiles": 5, - "complete_count": 2, + "complete_count": 3, "ready_to_run_count": 0, - "blocked_by_env_count": 3, + "blocked_by_env_count": 2, "blocked_by_preflight_count": 0, "missing_plan_count": 0, "planned_target_memories": 180000000, - "proven_target_memories": 60000000, + "proven_target_memories": 70000000, "nearest_baseline_max_memories": 10000000, "claim_status": "claims_limited" }, @@ -1690,8 +1682,8 @@ { "profile": "qdrant-sharded-10m", "requirement_id": "qdrant_sharded_10m_service", - "status": "blocked_by_env", - "strict_status": "action_required", + "status": "complete", + "strict_status": "pass", "plan_status": "action_required", "preflight_status": "action_required", "engine": "qdrant-sharded-service", @@ -1700,7 +1692,7 @@ "target_p99_ms": 100.0, "target_qps": 250.0, "output_artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", - "output_artifact_exists": false, + "output_artifact_exists": true, "checkpoint_path": "state/production-runs/qdrant-sharded-service-10000000.checkpoint.json", "missing_env": [ "WAVEMIND_QDRANT_URLS" @@ -1724,7 +1716,7 @@ }, "baseline_progress_ratio": 0.1, "target_gap_multiplier": 10.0, - "next_action": "Provision the listed environment, run the command, then promote the result artifact through the ingest gate." + "next_action": "Strict result artifact already passes." }, { "profile": "pgvector-10m", @@ -1855,8 +1847,8 @@ "readiness_status": "action_required", "claim_status": "claims_limited", "total_requirements": 8, - "action_required_count": 5, - "complete_count": 3, + "action_required_count": 4, + "complete_count": 4, "ready_for_safe_dispatch_count": 0, "can_auto_run_now_count": 0, "target_memories_total": 180000000, @@ -1864,12 +1856,12 @@ "pass": 8 }, "dispatch_status_counts": { - "blocked_by_preflight": 5, - "complete": 3 + "blocked_by_preflight": 4, + "complete": 4 }, "blocker_counts": { - "complete": 3, - "missing_env": 5 + "complete": 4, + "missing_env": 4 } }, "checks": [ @@ -2103,14 +2095,14 @@ { "id": "qdrant_sharded_10m_service", "title": "10M sharded Qdrant service load", - "strict_status": "action_required", + "strict_status": "pass", "preflight_status": "action_required", - "dispatch_status": "blocked_by_preflight", - "scale_gap_status": "blocked_by_env", + "dispatch_status": "complete", + "scale_gap_status": "complete", "workflow": "production-streaming-load.yml", "wave": "service-scale-10m", "artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", - "artifact_exists": false, + "artifact_exists": true, "output_artifact": "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json", "claim_unlocked": "Horizontally sharded Qdrant service recall/latency SLO.", "locked_claim": "10M-100M service-backed production scale", @@ -2140,8 +2132,8 @@ "post_ingest_refresh_command": "python benchmarks/strict_evidence_readiness_report.py --output benchmarks/strict_evidence_readiness_results.json --markdown-output benchmarks/STRICT_EVIDENCE_READINESS.md", "ready_for_safe_dispatch": false, "can_auto_run_now": false, - "next_action": "Provision the listed environment, run the command, then promote the result artifact through the ingest gate.", - "blocker_category": "missing_env" + "next_action": "Strict result artifact already passes.", + "blocker_category": "complete" }, { "id": "pgvector_10m_service", diff --git a/tests/test_benchmark_leaderboard.py b/tests/test_benchmark_leaderboard.py index 99cbeee..cad4911 100644 --- a/tests/test_benchmark_leaderboard.py +++ b/tests/test_benchmark_leaderboard.py @@ -62,7 +62,9 @@ def test_benchmark_leaderboard_renderer_writes_compact_leaderboard(tmp_path): assert "iterative recall `0.97`" in leaderboard assert "Qdrant streaming" in leaderboard assert "Qdrant sharded streaming" in leaderboard - assert "real two-service fanout smoke" in leaderboard + assert "real fanout smoke plus measured four-service 10M profile" in leaderboard + assert "10M recall `0.993`" in leaderboard + assert "10M p99 `71.3 ms`" in leaderboard assert "Qdrant 1M streaming" in leaderboard assert "tuned p99" in leaderboard assert "pgvector streaming" in leaderboard diff --git a/tests/test_benchmark_registry.py b/tests/test_benchmark_registry.py index 7590c5c..66036ef 100644 --- a/tests/test_benchmark_registry.py +++ b/tests/test_benchmark_registry.py @@ -149,6 +149,13 @@ def test_benchmark_matrix_contains_implemented_and_public_benchmarks(): assert qdrant_sharded_smoke["p99_latency_ms"] < 100.0 assert qdrant_sharded_smoke["slo_status"] == "pass" assert qdrant_sharded_smoke["shard_count"] == 2 + qdrant_sharded_10m = entries["production_streaming_load_runner"]["current"][ + "10M Qdrant sharded measured / Qdrant sharded service streaming" + ] + assert qdrant_sharded_10m["target_recall_at_k"] >= 0.95 + assert qdrant_sharded_10m["p99_latency_ms"] < 100.0 + assert qdrant_sharded_10m["shard_count"] == 4 + assert qdrant_sharded_10m["cost_status"] == "valid_slo" pgvector_smoke = entries["production_streaming_load_runner"]["current"][ "pgvector smoke / WaveMind pgvector streaming" ] diff --git a/tests/test_cost_efficiency_leaderboard.py b/tests/test_cost_efficiency_leaderboard.py index f59270a..00ea289 100644 --- a/tests/test_cost_efficiency_leaderboard.py +++ b/tests/test_cost_efficiency_leaderboard.py @@ -58,6 +58,17 @@ def test_cost_efficiency_leaderboard_renderer_writes_json_and_markdown(tmp_path) assert qdrant_10m["p99_latency_ms"] < 100.0 assert qdrant_10m["valid_cost"] is True + qdrant_sharded_10m = next( + row + for row in payload["measured_rows"] + if row["source_file"] + == "benchmarks/production_streaming_load_qdrant_sharded_10m_results.json" + ) + assert qdrant_sharded_10m["memory_count"] == 10_000_000 + assert qdrant_sharded_10m["recall_at_k"] >= 0.95 + assert qdrant_sharded_10m["p99_latency_ms"] < 100.0 + assert qdrant_sharded_10m["valid_cost"] is True + assert planned["faiss-ivfpq-50m"]["evidence_level"] == "planned" assert planned["faiss-ivfpq-50m"]["memory_count"] == 50_000_000 assert planned["faiss-ivfpq-50m"]["claim_status"] == "plan_only" diff --git a/tests/test_leaderboard_status.py b/tests/test_leaderboard_status.py index 5fadfa5..0e3c2d4 100644 --- a/tests/test_leaderboard_status.py +++ b/tests/test_leaderboard_status.py @@ -289,7 +289,7 @@ def test_leaderboard_status_renderer_writes_public_contract(tmp_path): "wavemind.production_evidence_bundle.v1" ) assert payload["production_evidence_bundle"]["claim_status"] == "claims_limited" - assert payload["production_evidence_bundle"]["next_action_count"] == 5 + assert payload["production_evidence_bundle"]["next_action_count"] == 4 assert payload["production_evidence_bundle"]["production_scale_run_contract"]["status"] == "available" assert payload["production_evidence_env"]["schema"] == ( "wavemind.production_evidence_env_contract.v1" @@ -460,7 +460,7 @@ def test_leaderboard_status_renderer_writes_public_contract(tmp_path): ) assert { "external_http_active_active", - "qdrant_sharded_10m_service", + "pgvector_10m_service", "hundred_million_remote_load", }.issubset( {entry["id"] for entry in payload["strict_production_evidence"]["action_required"]} diff --git a/tests/test_production_evidence_bundle.py b/tests/test_production_evidence_bundle.py index d7f1944..263d54c 100644 --- a/tests/test_production_evidence_bundle.py +++ b/tests/test_production_evidence_bundle.py @@ -53,8 +53,8 @@ def test_production_evidence_bundle_keeps_claims_limited_without_remote_artifact assert payload["summary"]["production_scale_run_contract_status"] == "available" assert payload["summary"]["production_scale_run_profile_count"] == 5 assert payload["summary"]["production_scale_run_target_memories_total"] == 180_000_000 - assert payload["summary"]["strict_pass_count"] == 3 - assert payload["summary"]["next_action_count"] == 5 + assert payload["summary"]["strict_pass_count"] == 4 + assert payload["summary"]["next_action_count"] == 4 assert payload["production_scale_run_contract"]["status"] == "available" assert payload["production_scale_run_contract"]["profile_count"] == 5 @@ -131,8 +131,8 @@ def test_scale_gap_manifest_tracks_large_n_proof_gaps(): assert payload["overall_status"] == "action_required" assert payload["summary"]["total_profiles"] == 5 assert payload["summary"]["planned_target_memories"] == 180_000_000 - assert payload["summary"]["complete_count"] == 2 - assert payload["summary"]["proven_target_memories"] == 60_000_000 + assert payload["summary"]["complete_count"] == 3 + assert payload["summary"]["proven_target_memories"] == 70_000_000 assert payload["summary"]["nearest_baseline_max_memories"] >= 10_000_000 gaps = {row["profile"]: row for row in payload["profile_gaps"]} @@ -149,6 +149,8 @@ def test_scale_gap_manifest_tracks_large_n_proof_gaps(): ) assert gaps["qdrant-10m"]["status"] == "complete" assert gaps["qdrant-10m"]["strict_status"] == "pass" + assert gaps["qdrant-sharded-10m"]["status"] == "complete" + assert gaps["qdrant-sharded-10m"]["strict_status"] == "pass" assert gaps["qdrant-10m"]["nearest_baseline"]["vectors"] >= 1_000_000 assert gaps["faiss-ivfpq-50m"]["nearest_baseline"]["vectors"] >= 10_000_000 assert gaps["faiss-ivfpq-50m"]["target_gap_multiplier"] == 5.0 diff --git a/tests/test_production_evidence_dispatch.py b/tests/test_production_evidence_dispatch.py index e535e0c..d9b49e2 100644 --- a/tests/test_production_evidence_dispatch.py +++ b/tests/test_production_evidence_dispatch.py @@ -51,9 +51,9 @@ def test_dispatch_plan_reports_blocked_jobs_without_remote_prerequisites(): assert payload["schema"] == "wavemind.production_evidence_dispatch.v1" assert payload["overall_status"] == "action_required" assert payload["summary"]["total_jobs"] == 8 - assert payload["summary"]["blocked_by_preflight_count"] == 5 + assert payload["summary"]["blocked_by_preflight_count"] == 4 assert payload["summary"]["ready_to_dispatch_count"] == 0 - assert payload["summary"]["complete_count"] == 3 + assert payload["summary"]["complete_count"] == 4 by_id = {row["id"]: row for row in payload["jobs"]} assert by_id["external_http_cluster"]["workflow"] == "external-http-cluster-load.yml" @@ -86,9 +86,9 @@ def test_dispatch_plan_becomes_ready_with_prerequisites_without_leaking_secret_v serialized = json.dumps(payload, sort_keys=True) assert payload["overall_status"] == "ready_to_dispatch" - assert payload["summary"]["ready_to_dispatch_count"] == 5 + assert payload["summary"]["ready_to_dispatch_count"] == 4 assert payload["summary"]["blocked_by_preflight_count"] == 0 - assert payload["summary"]["complete_count"] == 3 + assert payload["summary"]["complete_count"] == 4 assert payload["summary"]["runner_label"] == "self-hosted-xxl" assert "test-key" not in serialized diff --git a/tests/test_production_streaming_load_benchmark.py b/tests/test_production_streaming_load_benchmark.py index 80a8f34..63ed104 100644 --- a/tests/test_production_streaming_load_benchmark.py +++ b/tests/test_production_streaming_load_benchmark.py @@ -78,6 +78,7 @@ def test_streaming_load_qdrant_chunks_large_upsert_batches(): QdrantShardTarget, _chunks, _merge_scored_hits, + _iter_qdrant_shard_point_chunks, _qdrant_shard_index, _iter_qdrant_point_chunks, _upsert_qdrant_point_chunks, @@ -92,6 +93,34 @@ def test_streaming_load_qdrant_chunks_large_upsert_batches(): with pytest.raises(ValueError, match="shard_count must be positive"): _qdrant_shard_index(1, 0) + class Point: + def __init__(self, *, id, vector): + self.id = id + self.vector = vector + + shard_chunks = list( + _iter_qdrant_shard_point_chunks( + np.arange(1, 8, dtype=np.int64), + np.arange(14, dtype=np.float32).reshape(7, 2), + shard_index=1, + shard_count=3, + point_type=Point, + chunk_size=2, + ) + ) + assert [[point.id for point in chunk] for chunk in shard_chunks] == [[2, 5]] + with pytest.raises(ValueError, match="chunk size must be positive"): + list( + _iter_qdrant_shard_point_chunks( + np.array([1]), + np.array([[1.0]], dtype=np.float32), + shard_index=0, + shard_count=1, + point_type=Point, + chunk_size=0, + ) + ) + class Hit: def __init__(self, point_id, score): self.id = point_id @@ -122,8 +151,10 @@ def upsert(self, *, collection_name, points): executor=executor, clients=clients, targets=targets, - points_by_shard={0: [1, 3, 5], 1: [2, 4]}, - upsert_batch_size=2, + point_chunks_by_shard={ + 0: iter([[1, 3], [5]]), + 1: iter([[2, 4]]), + }, ) assert inserted == 5 @@ -474,6 +505,145 @@ def fail_if_vectors_are_regenerated(**kwargs): assert row["search_params"]["quantization"]["rescore"] is False +def test_qdrant_sharded_complete_resume_uses_grpc_and_quantization( + tmp_path, + monkeypatch, +): + from benchmarks import production_streaming_load_benchmark as benchmark + + class Model: + def __init__(self, **kwargs): + self.__dict__.update(kwargs) + + class FakeQdrantClient: + init_kwargs = [] + query_kwargs = [] + update_kwargs = [] + + def __init__(self, **kwargs): + type(self).init_kwargs.append(kwargs) + + def recreate_collection(self, **kwargs): + raise AssertionError("complete sharded checkpoint must reuse collections") + + def get_collection(self, *, collection_name): + return SimpleNamespace( + points_count=32, + indexed_vectors_count=32, + status="green", + optimizer_status="ok", + ) + + def update_collection(self, **kwargs): + type(self).update_kwargs.append(kwargs) + return True + + def query_points(self, **kwargs): + type(self).query_kwargs.append(kwargs) + return SimpleNamespace(points=[SimpleNamespace(id=1, score=1.0)]) + + def close(self): + return None + + fake_models = SimpleNamespace( + Distance=SimpleNamespace(COSINE="Cosine"), + HnswConfigDiff=Model, + OptimizersConfigDiff=Model, + PointStruct=Model, + QuantizationSearchParams=Model, + ScalarQuantization=Model, + ScalarQuantizationConfig=Model, + ScalarType=SimpleNamespace(INT8="int8"), + SearchParams=Model, + VectorParams=Model, + ) + monkeypatch.setitem( + sys.modules, + "qdrant_client", + SimpleNamespace(QdrantClient=FakeQdrantClient), + ) + monkeypatch.setitem(sys.modules, "qdrant_client.models", fake_models) + + checkpoint_path = tmp_path / "qdrant-sharded-complete.checkpoint.json" + urls = ["http://127.0.0.1:6335", "http://127.0.0.1:6337"] + collection_prefix = "complete_sharded_resume" + monkeypatch.setenv("WAVEMIND_QDRANT_URLS", ",".join(urls)) + monkeypatch.setenv("WAVEMIND_QDRANT_COLLECTION_PREFIX", collection_prefix) + monkeypatch.setenv("WAVEMIND_STREAMING_CHECKPOINT_PATH", str(checkpoint_path)) + monkeypatch.setenv("WAVEMIND_QDRANT_DEFER_INDEXING", "0") + monkeypatch.setenv("WAVEMIND_QDRANT_PREFER_GRPC", "1") + monkeypatch.setenv("WAVEMIND_QDRANT_GRPC_PORTS", "6336,6338") + monkeypatch.setenv("WAVEMIND_QDRANT_QUERY_TIMEOUT_SECONDS", "300") + monkeypatch.setenv("WAVEMIND_QDRANT_HNSW_ON_DISK", "0") + monkeypatch.setenv("WAVEMIND_QDRANT_SCALAR_QUANTIZATION", "1") + monkeypatch.setenv("WAVEMIND_QDRANT_SCALAR_QUANTILE", "0.99") + monkeypatch.setenv("WAVEMIND_QDRANT_SCALAR_ALWAYS_RAM", "1") + monkeypatch.setenv("WAVEMIND_QDRANT_QUANTIZATION_RESCORE", "0") + + source_ids = benchmark.choose_source_ids(64, 4, 3) + signature = benchmark._checkpoint_signature( + engine="Qdrant sharded service streaming", + count=64, + dim=8, + query_count=4, + top_k=2, + seed=3, + noise=0.01, + batch_size=32, + extra={ + "collection_config": benchmark._qdrant_collection_config_from_env(), + "target_urls": urls, + }, + ) + checkpoint = benchmark._new_checkpoint(signature) + checkpoint["metadata"]["collection_prefix"] = collection_prefix + checkpoint["completed_batch_starts"] = [1, 33] + checkpoint["source_vectors"] = { + str(source_id): [1.0] + [0.0] * 7 for source_id in source_ids + } + benchmark._write_checkpoint(checkpoint_path, checkpoint) + + def fail_if_vectors_are_regenerated(**kwargs): + raise AssertionError("complete sharded checkpoint must not regenerate batches") + + monkeypatch.setattr( + benchmark, + "iter_vector_batches", + fail_if_vectors_are_regenerated, + ) + row = benchmark.run_qdrant_sharded_streaming( + count=64, + dim=8, + query_count=4, + top_k=2, + seed=3, + noise=0.01, + batch_size=32, + ) + + assert row["qdrant_checkpoint_complete_resume"] is True + assert row["checkpoint_completed_batches"] == 2 + assert row["checkpoint_source_vectors"] == 4 + assert row["transport"] == { + "prefer_grpc": True, + "grpc_ports": [6336, 6338], + "query_timeout_seconds": 300, + } + assert [call["grpc_port"] for call in FakeQdrantClient.init_kwargs] == [ + 6336, + 6338, + ] + assert all(call["prefer_grpc"] is True for call in FakeQdrantClient.init_kwargs) + assert {call["timeout"] for call in FakeQdrantClient.query_kwargs} == {300} + assert len(FakeQdrantClient.update_kwargs) == 2 + assert all( + call["quantization_config"].scalar.always_ram is True + for call in FakeQdrantClient.update_kwargs + ) + assert row["search_params"]["quantization"]["rescore"] is False + assert row["collection_params"]["scalar_quantization"]["quantile"] == 0.99 + + def test_streaming_load_faiss_ivfpq_smoke(tmp_path, monkeypatch): pytest.importorskip("faiss") @@ -1032,6 +1202,11 @@ def test_streaming_load_plan_only_supports_qdrant_sharded_service(monkeypatch): assert "WAVEMIND_QDRANT_URLS" in row["required_env"] assert "missing_env:WAVEMIND_QDRANT_URLS" in row["blockers"] assert row["command_env"]["WAVEMIND_QDRANT_FANOUT_WORKERS"] == "6" + assert row["command_env"]["WAVEMIND_QDRANT_PREFER_GRPC"] == "1" + assert row["command_env"]["WAVEMIND_QDRANT_GRPC_PORT"] == "6334" + assert row["command_env"]["WAVEMIND_QDRANT_HNSW_ON_DISK"] == "0" + assert row["command_env"]["WAVEMIND_QDRANT_SCALAR_QUANTIZATION"] == "1" + assert row["command_env"]["WAVEMIND_QDRANT_QUANTIZATION_RESCORE"] == "0" assert row["command_env"]["WAVEMIND_QDRANT_REQUIRE_FULL_INDEX"] == "1" assert row["command_env"]["WAVEMIND_QDRANT_DEFER_INDEXING"] == "1" assert ( diff --git a/tests/test_strict_evidence_readiness_report.py b/tests/test_strict_evidence_readiness_report.py index c1e51d7..56757d2 100644 --- a/tests/test_strict_evidence_readiness_report.py +++ b/tests/test_strict_evidence_readiness_report.py @@ -18,13 +18,13 @@ def test_strict_evidence_readiness_joins_all_strict_requirements(): assert payload["readiness_status"] == "action_required" assert payload["claim_status"] == "claims_limited" assert payload["summary"]["total_requirements"] == 8 - assert payload["summary"]["action_required_count"] == 5 - assert payload["summary"]["complete_count"] == 3 + assert payload["summary"]["action_required_count"] == 4 + assert payload["summary"]["complete_count"] == 4 assert payload["summary"]["target_memories_total"] == 180_000_000 assert payload["summary"]["check_counts"] == {"pass": 8} assert payload["summary"]["can_auto_run_now_count"] == 0 - assert payload["summary"]["blocker_counts"]["missing_env"] == 5 - assert payload["summary"]["blocker_counts"]["complete"] == 3 + assert payload["summary"]["blocker_counts"]["missing_env"] == 4 + assert payload["summary"]["blocker_counts"]["complete"] == 4 by_id = {row["id"]: row for row in payload["requirements"]} assert set(by_id) == {