feat(search): default to potion-code-16M-v2, invalidate stale-model caches

.claude/skills/verify/SKILL.md

@@ -0,0 +1,17 @@
+---
+name: verify
+summary: Exercise csp indexing, persisted-index search, and cache reuse through the built CLI.
+---
+
+# Verify csp
+
+1. Build the user-facing binary with `cargo build -p code-search-please --bin csp`.
+2. Use a scratch directory outside the repository for all generated indexes and HOME state.
+3. Drive persisted-index behavior:
+   - `target/debug/csp index crates/csp/src --out <scratch>/index`
+   - `target/debug/csp search "load_or_build_index cache" --index <scratch>/index --top-k 3`
+4. Drive auto-cache behavior twice with an isolated home:
+   - `HOME=<scratch>/home target/debug/csp search "DEFAULT_MODEL_NAME" crates/csp/src/indexing --top-k 1`
+   - Repeat the same command to exercise cache reuse.
+5. Probe errors with a missing `--index` path; expect a clear `Index not found` message and exit status 1.
+6. Do not use tests as runtime verification evidence; run the project quality gate separately.

.please/docs/references/semble.md

@@ -338,7 +338,7 @@ Clean two-layer split:
 | strong / moderate / mild penalty | `0.3` / `0.5` / `0.7` | same | `ranking/penalties.rs` |
 | file saturation threshold / decay | `1` / `0.5` | `1` / `0.5` | `ranking/penalties.rs` |
 | max file bytes | `1_000_000` | `1_000_000` | `index/files.py` / `indexing/create.rs` |
-| default model | `minishlab/potion-code-16M` | same (real + stub) | `utils.py` / `indexing/dense.rs` |
+| default model | `minishlab/potion-code-16M-v2` (semble#219) | same (real + stub); cache keyed on model_id | `utils.py` / `indexing/dense.rs` |
 | MCP in-mem LRU | `10` | `10` | `mcp.py` / `csp::mcp` |
 | cache dir mode | — | `0o700` | `indexing/cache.rs` |
 

README.ko.md

@@ -497,7 +497,7 @@ for (const { chunk, score } of results) {
 
 ## 동작 원리
 
-`csp`는 [tree-sitter](https://tree-sitter.github.io/)로 각 파일을 코드 인지형 청크로 분할한 뒤, 두 개의 상호 보완적인 검색기로 쿼리를 모든 청크와 점수화합니다. 의미 유사도를 위한 정적 [Model2Vec](https://github.com/MinishLab/model2vec) 임베딩(코드 특화 `potion-code-16M` 모델 사용)과, 식별자/API명 등 어휘 매칭을 위한 BM25입니다. 두 점수 리스트는 Reciprocal Rank Fusion(RRF)으로 결합됩니다.
+`csp`는 [tree-sitter](https://tree-sitter.github.io/)로 각 파일을 코드 인지형 청크로 분할한 뒤, 두 개의 상호 보완적인 검색기로 쿼리를 모든 청크와 점수화합니다. 의미 유사도를 위한 정적 [Model2Vec](https://github.com/MinishLab/model2vec) 임베딩(코드 특화 `potion-code-16M-v2` 모델 사용)과, 식별자/API명 등 어휘 매칭을 위한 BM25입니다. 두 점수 리스트는 Reciprocal Rank Fusion(RRF)으로 결합됩니다.
 
 결합 후에는 코드 인지형 신호들로 결과를 재정렬합니다.
 

README.md

@@ -497,7 +497,7 @@ for (const { chunk, score } of results) {
 
 ## How it works
 
-`csp` splits each file into code-aware chunks using [tree-sitter](https://tree-sitter.github.io/), then scores every query against the chunks with two complementary retrievers: static [Model2Vec](https://github.com/MinishLab/model2vec) embeddings using the code-specialized `potion-code-16M` model for semantic similarity, and BM25 for lexical matches on identifiers and API names. The two score lists are fused with Reciprocal Rank Fusion (RRF).
+`csp` splits each file into code-aware chunks using [tree-sitter](https://tree-sitter.github.io/), then scores every query against the chunks with two complementary retrievers: static [Model2Vec](https://github.com/MinishLab/model2vec) embeddings using the code-specialized `potion-code-16M-v2` model for semantic similarity, and BM25 for lexical matches on identifiers and API names. The two score lists are fused with Reciprocal Rank Fusion (RRF).
 
 After fusing, results are reranked with a set of code-aware signals:
 

crates/csp/src/indexing/cache.rs

@@ -179,13 +179,33 @@ pub fn compute_content_hash(files: &[CacheFile]) -> String {
 
     let mut hasher = Sha256::new();
     for file in sorted {
-        let len16 = file.path.encode_utf16().count();
-        hasher.update(format!("{len16}:{}", file.path).as_bytes());
-        hasher.update(&file.content);
+        update_content_hash(&mut hasher, &file.path, &file.content);
     }
     to_hex(&hasher.finalize())
 }
 
+/// Compute the same content hash while reading one file at a time, avoiding
+/// retention of every file body for large repositories. Unreadable files are
+/// skipped, matching the indexer's existing best-effort collection behavior.
+pub(crate) fn compute_content_hash_from_paths(mut files: Vec<(String, PathBuf)>) -> String {
+    files.sort_by(|a, b| a.0.cmp(&b.0));
+
+    let mut hasher = Sha256::new();
+    for (relative_path, path) in files {
+        let Ok(content) = std::fs::read(path) else {
+            continue;
+        };
+        update_content_hash(&mut hasher, &relative_path, &content);
+    }
+    to_hex(&hasher.finalize())
+}
+
+fn update_content_hash(hasher: &mut Sha256, path: &str, content: &[u8]) {
+    let len16 = path.encode_utf16().count();
+    hasher.update(format!("{len16}:{path}").as_bytes());
+    hasher.update(content);
+}
+
 /// Directories from `home` down to `leaf` (inclusive), home-first. When `leaf`
 /// is not under `home`, only `leaf` is returned.
 fn chain_to(leaf: &Path, home: &Path) -> Vec<PathBuf> {
@@ -380,6 +400,22 @@ mod tests {
         assert_eq!(a, b);
     }
 
+    #[test]
+    fn streamed_content_hash_matches_in_memory_hash() {
+        let dir = tempdir().unwrap();
+        let a_path = dir.path().join("a.ts");
+        let b_path = dir.path().join("b.ts");
+        std::fs::write(&a_path, "one").unwrap();
+        std::fs::write(&b_path, "two").unwrap();
+
+        let streamed = compute_content_hash_from_paths(vec![
+            ("b.ts".to_string(), b_path),
+            ("a.ts".to_string(), a_path),
+        ]);
+        let in_memory = compute_content_hash(&[cfile("a.ts", "one"), cfile("b.ts", "two")]);
+        assert_eq!(streamed, in_memory);
+    }
+
     #[test]
     fn content_hash_is_hex_sha256() {
         let h = compute_content_hash(&[cfile("a.ts", "x")]);

crates/csp/src/indexing/cache_orchestrator.rs

@@ -0,0 +1,139 @@
+//! Build-or-reuse orchestration for the global on-disk index cache. Port of
+//! `src/indexing/cache.ts`.
+
+use std::path::{Path, PathBuf};
+
+use crate::chunking::source::DESIRED_CHUNK_LENGTH_CHARS;
+use crate::indexing::cache::{
+    compute_content_hash_from_paths, ensure_cache_dir, resolve_cache_dir, CacheLocation,
+};
+use crate::indexing::create::MAX_FILE_BYTES;
+use crate::indexing::dense::DEFAULT_MODEL_NAME;
+use crate::indexing::file_walker::walk_files;
+use crate::indexing::files::get_extensions;
+use crate::indexing::index::{normalize_content, parse_manifest, CspIndex, LoadOptions};
+use crate::types::ContentType;
+use crate::utils::is_git_url;
+
+/// Options for [`load_or_build_index`].
+#[derive(Debug, Clone, Default)]
+pub struct LoadOrBuildOptions {
+    pub base_dir: Option<PathBuf>,
+    pub git_ref: Option<String>,
+    pub content: Option<Vec<ContentType>>,
+    pub model_path: Option<String>,
+}
+
+fn normalized_hash_path(path: &Path) -> String {
+    path.to_string_lossy().replace('\\', "/")
+}
+
+/// Collect the source files `from_path` would index as sorted hash inputs.
+/// File contents are read later, one at a time, by the hashing helper.
+fn collect_source_paths(root: &Path, content: &[ContentType]) -> Vec<(String, PathBuf)> {
+    let resolved = get_extensions(content, None);
+    let ext_refs: Vec<&str> = resolved.iter().map(String::as_str).collect();
+    let mut files = Vec::new();
+    for file_path in walk_files(root, &ext_refs, &[]) {
+        let Ok(meta) = std::fs::metadata(&file_path) else {
+            continue;
+        };
+        if meta.len() > MAX_FILE_BYTES {
+            continue;
+        }
+        let rel = file_path.strip_prefix(root).unwrap_or(&file_path);
+        files.push((normalized_hash_path(rel), file_path));
+    }
+    files
+}
+
+/// Load a cached index for `source` if fresh, else build, persist, and return.
+pub fn load_or_build_index(source: &str, options: &LoadOrBuildOptions) -> Result<CspIndex, String> {
+    let content = normalize_content(options.content.clone());
+    let is_git = is_git_url(source);
+
+    let location = CacheLocation {
+        base_dir: options.base_dir.clone(),
+        git_ref: options.git_ref.clone(),
+    };
+    let cache_dir = resolve_cache_dir(source, &content, &location);
+    let base_only = CacheLocation {
+        base_dir: options.base_dir.clone(),
+        git_ref: None,
+    };
+    ensure_cache_dir(&cache_dir, &base_only)?;
+
+    // Local sources: the source-file hash is the cache-validity oracle. Git
+    // sources are URL+ref keyed (no cheap live hash).
+    let source_hash = if is_git {
+        None
+    } else {
+        Some(compute_content_hash_from_paths(collect_source_paths(
+            Path::new(source),
+            &content,
+        )))
+    };
+
+    // The resolved model name the index would be (re)built with. A cache built
+    // with a different model must not be reused — its vectors are incompatible
+    // (mirrors semble#219 bumping the default to the `-v2` weights).
+    let expected_model = options.model_path.as_deref().unwrap_or(DEFAULT_MODEL_NAME);
+
+    if let Some(cached) = try_reuse(&cache_dir, is_git, source_hash.as_deref(), expected_model) {
+        return Ok(cached);
+    }
+
+    let load_options = LoadOptions {
+        model_path: options.model_path.clone(),
+        content: Some(content),
+    };
+    let index = if is_git {
+        CspIndex::from_git(source, &load_options, options.git_ref.as_deref())?
+    } else {
+        CspIndex::from_path(Path::new(source), &load_options)?
+    };
+    index.save(&cache_dir, source_hash.as_deref())?;
+    Ok(index)
+}
+
+/// Reuse a cached index when present and valid, else `None`.
+fn try_reuse(
+    cache_dir: &Path,
+    is_git: bool,
+    source_hash: Option<&str>,
+    expected_model: &str,
+) -> Option<CspIndex> {
+    let manifest_path = cache_dir.join("manifest.json");
+    if !manifest_path.exists() {
+        return None;
+    }
+    let raw = std::fs::read_to_string(&manifest_path).ok()?;
+    let value: serde_json::Value = serde_json::from_str(&raw).ok()?;
+    let manifest = parse_manifest(&value).ok()?;
+    // A chunk_size change re-chunks every file, so a cache built with a different
+    // target length is stale even if the source files are byte-identical.
+    if manifest.chunk_size != Some(DESIRED_CHUNK_LENGTH_CHARS as u32) {
+        return None;
+    }
+    // A model change makes the persisted vectors incompatible with queries
+    // embedded by the new model, so a cache built with a different model is stale.
+    if manifest.model_id != expected_model {
+        return None;
+    }
+    // The persisted vectors and live query model must use the same runtime
+    // implementation. This distinguishes a real Model2Vec cache from an offline
+    // deterministic-stub cache even when both share the same requested model id.
+    let (query_model, _) = crate::indexing::dense::load_model(Some(expected_model));
+    if manifest.model_kind.as_deref() != Some(query_model.kind()) {
+        return None;
+    }
+    // Local sources additionally validate the live source-file hash; git sources
+    // are URL+ref keyed (no cheap live hash).
+    if !is_git && Some(manifest.content_hash.as_str()) != source_hash {
+        return None;
+    }
+    CspIndex::load_from_disk(cache_dir).ok()
+}
+
+#[cfg(test)]
+mod tests;

crates/csp/src/indexing/cache_orchestrator/tests.rs

@@ -0,0 +1,147 @@
+use super::*;
+use crate::indexing::cache::{resolve_cache_dir, CacheLocation};
+use crate::indexing::dense::{make_stub_model, SelectableBasicBackend};
+use crate::indexing::index::{CspIndexState, DEFAULT_CONTENT};
+use crate::indexing::sparse::Bm25Index;
+use crate::types::Chunk;
+use tempfile::tempdir;
+
+fn make_chunk(file_path: &str, content: &str) -> Chunk {
+    Chunk {
+        content: content.to_string(),
+        file_path: file_path.to_string(),
+        start_line: 1,
+        end_line: 10,
+        language: Some("typescript".to_string()),
+    }
+}
+
+fn build_index(chunks: Vec<Chunk>) -> CspIndex {
+    let model = make_stub_model(4);
+    let vectors = vec![vec![1.0, 0.0, 0.0, 0.0]; chunks.len()];
+    CspIndex::new(CspIndexState {
+        model,
+        bm25_index: Bm25Index::build(&vec![vec!["x".to_string()]; chunks.len()]),
+        semantic_index: SelectableBasicBackend::from_vectors(vectors).unwrap(),
+        chunks,
+        model_path: "test-model".to_string(),
+        root: None,
+        content: DEFAULT_CONTENT.to_vec(),
+    })
+}
+
+#[test]
+fn hash_path_normalizes_platform_separators() {
+    assert_eq!(normalized_hash_path(Path::new(r"src\lib.rs")), "src/lib.rs");
+    assert_eq!(normalized_hash_path(Path::new("src/lib.rs")), "src/lib.rs");
+}
+
+#[test]
+fn try_reuse_rejects_stale_chunk_size() {
+    let chunks = vec![make_chunk("a.ts", "A")];
+    let idx = build_index(chunks);
+    let dir = tempdir().unwrap();
+    idx.save(dir.path(), Some("deadbeef")).unwrap();
+
+    // Fresh cache (matching hash + current chunk_size) is reused.
+    assert!(try_reuse(dir.path(), false, Some("deadbeef"), "test-model").is_some());
+
+    // Rewrite the manifest with a different chunk_size → stale → rebuild.
+    let manifest_path = dir.path().join("manifest.json");
+    let raw = std::fs::read_to_string(&manifest_path).unwrap();
+    let mut value: serde_json::Value = serde_json::from_str(&raw).unwrap();
+    value["chunkSize"] = serde_json::json!(9999);
+    std::fs::write(&manifest_path, value.to_string()).unwrap();
+    assert!(try_reuse(dir.path(), false, Some("deadbeef"), "test-model").is_none());
+
+    // A manifest predating the field (absent chunkSize) is also stale.
+    let mut value: serde_json::Value = serde_json::from_str(&raw).unwrap();
+    value.as_object_mut().unwrap().remove("chunkSize");
+    std::fs::write(&manifest_path, value.to_string()).unwrap();
+    assert!(try_reuse(dir.path(), false, Some("deadbeef"), "test-model").is_none());
+}
+
+#[test]
+fn try_reuse_rejects_mismatched_model_kind() {
+    let idx = build_index(vec![make_chunk("a.ts", "A")]);
+    let dir = tempdir().unwrap();
+    idx.save(dir.path(), Some("deadbeef")).unwrap();
+
+    let manifest_path = dir.path().join("manifest.json");
+    let raw = std::fs::read_to_string(&manifest_path).unwrap();
+    let mut value: serde_json::Value = serde_json::from_str(&raw).unwrap();
+    value["modelKind"] = serde_json::json!("static");
+    std::fs::write(&manifest_path, value.to_string()).unwrap();
+
+    assert!(try_reuse(dir.path(), false, Some("deadbeef"), "test-model").is_none());
+}
+
+#[test]
+fn try_reuse_rejects_stale_model() {
+    let chunks = vec![make_chunk("a.ts", "A")];
+    let idx = build_index(chunks);
+    let dir = tempdir().unwrap();
+    idx.save(dir.path(), Some("deadbeef")).unwrap();
+
+    // Same model → reused; a different model (e.g. after a default bump to
+    // `-v2`) → stale even with a matching hash + chunk_size.
+    assert!(try_reuse(dir.path(), false, Some("deadbeef"), "test-model").is_some());
+    assert!(try_reuse(dir.path(), false, Some("deadbeef"), "other-model").is_none());
+}
+
+// --- load_or_build_index (cache.ts loadOrBuildIndex parity) ---
+
+#[test]
+fn load_or_build_miss_then_hit_then_invalidate() {
+    let home = tempdir().unwrap();
+    let src = tempdir().unwrap();
+    let base = home.path().join(".csp");
+    std::fs::write(
+        src.path().join("a.ts"),
+        "export function alpha() { return 1 }\n",
+    )
+    .unwrap();
+    let src_str = src.path().to_string_lossy().into_owned();
+    let opts = LoadOrBuildOptions {
+        base_dir: Some(base.clone()),
+        ..Default::default()
+    };
+
+    // Miss: builds and writes a manifest.
+    let first = load_or_build_index(&src_str, &opts).unwrap();
+    assert!(!first.chunks.is_empty());
+    let cache_dir = resolve_cache_dir(
+        &src_str,
+        DEFAULT_CONTENT,
+        &CacheLocation {
+            base_dir: Some(base.clone()),
+            git_ref: None,
+        },
+    );
+    let manifest_path = cache_dir.join("manifest.json");
+    assert!(manifest_path.exists());
+
+    // Add an ignored sentinel to make reuse observable: a rebuild rewrites the
+    // manifest from the typed struct and removes this extra field.
+    let raw = std::fs::read_to_string(&manifest_path).unwrap();
+    let mut manifest: serde_json::Value = serde_json::from_str(&raw).unwrap();
+    manifest["reuseSentinel"] = serde_json::json!(true);
+    std::fs::write(&manifest_path, manifest.to_string()).unwrap();
+
+    // Hit: a second call reuses the cache without rewriting the manifest.
+    let second = load_or_build_index(&src_str, &opts).unwrap();
+    assert_eq!(second.chunks.len(), first.chunks.len());
+    let after_hit: serde_json::Value =
+        serde_json::from_str(&std::fs::read_to_string(&manifest_path).unwrap()).unwrap();
+    assert_eq!(after_hit["reuseSentinel"], serde_json::json!(true));
+
+    // Invalidation: add a file → content hash changes → rebuild reflects it.
+    std::fs::write(
+        src.path().join("b.ts"),
+        "export function beta() { return 2 }\n",
+    )
+    .unwrap();
+    let third = load_or_build_index(&src_str, &opts).unwrap();
+    assert!(third.chunks.iter().any(|c| c.file_path == "b.ts"));
+    assert!(third.chunks.len() >= first.chunks.len());
+}