diff --git a/README.md b/README.md index cf5bc739..d54438cf 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ ## Table of Contents +- [What's new (2026-06-21) — Full-Text Search (BM25)](#whats-new-2026-06-21--full-text-search-bm25) - [What's new (2026-06-21) — JSON Pointer, Patch & Merge Patch](#whats-new-2026-06-21--json-pointer-patch--merge-patch) - [What's new (2026-06-21) — Client-Side Rate Limiting](#whats-new-2026-06-21--client-side-rate-limiting) - [What's new (2026-06-21) — JSON Web Tokens (JWT)](#whats-new-2026-06-21--json-web-tokens-jwt) @@ -116,6 +117,12 @@ --- +## What's new (2026-06-21) — Full-Text Search (BM25) + +Rank a document corpus by relevance. Full reference: [`docs/source/Eng/doc/new_features/v64_features_doc.rst`](docs/source/Eng/doc/new_features/v64_features_doc.rst). + +- **`SearchIndex` / `search_documents` / `tokenize`** (`AC_search_documents`, `ac_search_documents`): `fuzzy` is pairwise and `skill_library` matches substrings alphabetically — neither ranks a corpus by relevance. This adds an inverted-index search ranked with Okapi BM25 (`k1=1.5`, `b=0.75`, `IDF = ln(1+(N−df+0.5)/(df+0.5))`) or TF-IDF, so a rare term out-ranks a common one, term frequency saturates, and long docs are normalized down. Incremental `add`/`remove`, optional stop-words, deterministic ranking. Pure-stdlib `math`+`collections`+`re` — no database. + ## What's new (2026-06-21) — JSON Pointer, Patch & Merge Patch Address, diff and patch JSON. Full reference: [`docs/source/Eng/doc/new_features/v63_features_doc.rst`](docs/source/Eng/doc/new_features/v63_features_doc.rst). diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index f1ca0c03..901ecc28 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -12,6 +12,7 @@ ## 目录 +- [本次更新 (2026-06-21) — 全文搜索(BM25)](#本次更新-2026-06-21--全文搜索bm25) - [本次更新 (2026-06-21) — JSON Pointer、Patch 与 Merge Patch](#本次更新-2026-06-21--json-pointerpatch-与-merge-patch) - [本次更新 (2026-06-21) — 客户端速率限制](#本次更新-2026-06-21--客户端速率限制) - [本次更新 (2026-06-21) — JSON Web Token(JWT)](#本次更新-2026-06-21--json-web-tokenjwt) @@ -115,6 +116,12 @@ --- +## 本次更新 (2026-06-21) — 全文搜索(BM25) + +依相关性对文档语料排名。完整参考:[`docs/source/Zh/doc/new_features/v64_features_doc.rst`](../docs/source/Zh/doc/new_features/v64_features_doc.rst)。 + +- **`SearchIndex` / `search_documents` / `tokenize`**(`AC_search_documents`、`ac_search_documents`):`fuzzy` 是成对的,`skill_library` 以字母序匹配子字符串 —— 两者都不会依相关性对语料排名。本功能补上以倒排索引、用 Okapi BM25(`k1=1.5`、`b=0.75`、`IDF = ln(1+(N−df+0.5)/(df+0.5))`)或 TF-IDF 排名的搜索,因此罕见词胜过常见词、词频会饱和、长文档被规范化下调。增量 `add`/`remove`、可选停用词、结果确定。纯标准库 `math`+`collections`+`re` —— 无需数据库。 + ## 本次更新 (2026-06-21) — JSON Pointer、Patch 与 Merge Patch 定址、取差异并修补 JSON。完整参考:[`docs/source/Zh/doc/new_features/v63_features_doc.rst`](../docs/source/Zh/doc/new_features/v63_features_doc.rst)。 diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index 0f66374c..90c6a853 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -12,6 +12,7 @@ ## 目錄 +- [本次更新 (2026-06-21) — 全文搜尋(BM25)](#本次更新-2026-06-21--全文搜尋bm25) - [本次更新 (2026-06-21) — JSON Pointer、Patch 與 Merge Patch](#本次更新-2026-06-21--json-pointerpatch-與-merge-patch) - [本次更新 (2026-06-21) — 用戶端速率限制](#本次更新-2026-06-21--用戶端速率限制) - [本次更新 (2026-06-21) — JSON Web Token(JWT)](#本次更新-2026-06-21--json-web-tokenjwt) @@ -115,6 +116,12 @@ --- +## 本次更新 (2026-06-21) — 全文搜尋(BM25) + +依相關性對文件語料排名。完整參考:[`docs/source/Zh/doc/new_features/v64_features_doc.rst`](../docs/source/Zh/doc/new_features/v64_features_doc.rst)。 + +- **`SearchIndex` / `search_documents` / `tokenize`**(`AC_search_documents`、`ac_search_documents`):`fuzzy` 是成對的,`skill_library` 以字母序比對子字串 —— 兩者都不會依相關性對語料排名。本功能補上以倒排索引、用 Okapi BM25(`k1=1.5`、`b=0.75`、`IDF = ln(1+(N−df+0.5)/(df+0.5))`)或 TF-IDF 排名的搜尋,因此罕見詞勝過常見詞、詞頻會飽和、長文件被正規化下調。增量 `add`/`remove`、可選停用詞、結果具決定性。純標準函式庫 `math`+`collections`+`re` —— 無需資料庫。 + ## 本次更新 (2026-06-21) — JSON Pointer、Patch 與 Merge Patch 定址、取差異並修補 JSON。完整參考:[`docs/source/Zh/doc/new_features/v63_features_doc.rst`](../docs/source/Zh/doc/new_features/v63_features_doc.rst)。 diff --git a/docs/source/Eng/doc/new_features/v64_features_doc.rst b/docs/source/Eng/doc/new_features/v64_features_doc.rst new file mode 100644 index 00000000..02994241 --- /dev/null +++ b/docs/source/Eng/doc/new_features/v64_features_doc.rst @@ -0,0 +1,49 @@ +Full-Text Search (BM25) +======================= + +``fuzzy`` does pairwise string similarity and ``skill_library`` does +alphabetical substring matching, but neither ranks a *corpus* of documents by +relevance — a rare distinctive term and a ubiquitous one weigh the same. This +adds an inverted-index search that ranks documents with Okapi **BM25** (or +TF-IDF), so flows and agents can search logs, scraped records, or knowledge +snippets without a database. + +Pure standard library (``math`` + ``collections`` + ``re``); deterministic; +imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import SearchIndex, search_documents + + corpus = { + "d1": "the quick brown fox jumps over the lazy dog", + "d2": "a quick brown dog runs fast", + "d3": "the database stores quick query results", + } + + index = SearchIndex.build(corpus) # or SearchIndex(); index.add(id, text) + for hit in index.search("quick dog", top_k=5): + print(hit.doc_id, hit.score) + + # one-shot convenience + hits = search_documents(corpus, "database", mode="bm25") + +``SearchIndex.add`` / ``remove`` keep the index up to date incrementally; +``build`` indexes a ``{doc_id: text}`` map (or ``(id, text)`` pairs). ``search`` +returns ranked ``SearchHit(doc_id, score)`` results — by default BM25 +(``k1=1.5``, ``b=0.75``), or ``mode="tfidf"``. The scoring is the standard +Okapi formula with ``IDF = ln(1 + (N − df + 0.5) / (df + 0.5))``, so a rare term +out-ranks a common one, term-frequency saturates (``k1``), and long documents +are normalized down (``b``). A ``stop_words`` set can be supplied to drop noise +terms. Results are deterministic (ties broken by ``doc_id``). + +Executor command +---------------- + +``AC_search_documents`` takes ``docs`` (a ``{doc_id: text}`` map or JSON +string), a ``query``, and optional ``top_k`` / ``mode``; it returns +``{hits: [{doc_id, score}]}``. The same operation is exposed as the MCP tool +``ac_search_documents`` and as a Script Builder command under **Data**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index cc695a6f..bae93915 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -86,6 +86,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v61_features_doc doc/new_features/v62_features_doc doc/new_features/v63_features_doc + doc/new_features/v64_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v64_features_doc.rst b/docs/source/Zh/doc/new_features/v64_features_doc.rst new file mode 100644 index 00000000..730e272c --- /dev/null +++ b/docs/source/Zh/doc/new_features/v64_features_doc.rst @@ -0,0 +1,42 @@ +全文搜尋(BM25) +=============== + +``fuzzy`` 做成對的字串相似度,``skill_library`` 做字母序的子字串比對,但兩者都無法依相關性對 +*文件語料*排名 —— 罕見的獨特詞與隨處可見的詞權重相同。本功能補上以倒排索引、用 Okapi **BM25** +(或 TF-IDF)排名文件的搜尋,讓流程與 agent 不需資料庫即可搜尋日誌、爬取紀錄或知識片段。 + +純標準函式庫(``math`` + ``collections`` + ``re``);具決定性;不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import SearchIndex, search_documents + + corpus = { + "d1": "the quick brown fox jumps over the lazy dog", + "d2": "a quick brown dog runs fast", + "d3": "the database stores quick query results", + } + + index = SearchIndex.build(corpus) # 或 SearchIndex(); index.add(id, text) + for hit in index.search("quick dog", top_k=5): + print(hit.doc_id, hit.score) + + # 一次性便利函式 + hits = search_documents(corpus, "database", mode="bm25") + +``SearchIndex.add`` / ``remove`` 以增量方式維護索引;``build`` 索引一個 ``{doc_id: text}`` 對映 +(或 ``(id, text)`` 配對)。``search`` 回傳排名後的 ``SearchHit(doc_id, score)`` 結果 —— 預設 +為 BM25(``k1=1.5``、``b=0.75``),或 ``mode="tfidf"``。評分採標準 Okapi 公式, +``IDF = ln(1 + (N − df + 0.5) / (df + 0.5))``,因此罕見詞勝過常見詞、詞頻會飽和(``k1``)、長 +文件被正規化下調(``b``)。可提供 ``stop_words`` 集合以濾除雜訊詞。結果具決定性(平手以 +``doc_id`` 決定)。 + +執行器命令 +---------- + +``AC_search_documents`` 接受 ``docs``(``{doc_id: text}`` 對映或 JSON 字串)、``query`` 與選用 +的 ``top_k`` / ``mode``,回傳 ``{hits: [{doc_id, score}]}``。同一操作亦以 MCP 工具 +``ac_search_documents`` 以及 Script Builder 中 **Data** 分類下的命令提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index 1500da46..fbfcf6ce 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -86,6 +86,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v61_features_doc doc/new_features/v62_features_doc doc/new_features/v63_features_doc + doc/new_features/v64_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index d0024b38..dc970452 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -329,6 +329,10 @@ PatchError, PatchTestFailed, apply_patch, make_merge_patch, make_patch, merge_patch, remove_pointer, resolve_pointer, set_pointer, ) +# In-memory BM25 / TF-IDF full-text search +from je_auto_control.utils.search_index import ( + SearchHit, SearchIndex, search_documents, tokenize, +) # Background popup/interrupt watchdog (unattended automation) from je_auto_control.utils.watchdog import ( PopupWatchdog, WatchdogRule, default_popup_watchdog, @@ -811,6 +815,7 @@ def start_autocontrol_gui(*args, **kwargs): "PatchError", "PatchTestFailed", "apply_patch", "make_merge_patch", "make_patch", "merge_patch", "remove_pointer", "resolve_pointer", "set_pointer", + "SearchHit", "SearchIndex", "search_documents", "tokenize", # MCP server "AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt", "MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 4b814ba9..ca027b37 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -1155,6 +1155,18 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None: ), description="Validate JSON against a JSON Schema; returns {ok, errors}.", )) + specs.append(CommandSpec( + "AC_search_documents", "Data", "Full-Text Search (BM25)", + fields=( + FieldSpec("docs", FieldType.STRING, + placeholder='{"d1": "quick brown fox", "d2": "lazy dog"}'), + FieldSpec("query", FieldType.STRING, placeholder="quick fox"), + FieldSpec("top_k", FieldType.INT, optional=True, default=10), + FieldSpec("mode", FieldType.STRING, optional=True, + placeholder="bm25", choices=("bm25", "tfidf")), + ), + description="Rank a {id: text} corpus for a query; returns {hits}.", + )) specs.append(CommandSpec( "AC_resolve_pointer", "Data", "JSON Pointer: Resolve", fields=( diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index e50249b1..6e7c6f57 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -2928,6 +2928,17 @@ def _rate_limit(name: str, rate: float = 1.0, capacity: float = 1.0, "wait": round(bucket.time_until_available(float(n)), 4)} +def _search_documents(docs: Any, query: str, top_k: int = 10, + mode: str = "bm25") -> Dict[str, Any]: + """Adapter: BM25/TF-IDF search a {doc_id: text} corpus (dict or JSON str).""" + import json + from je_auto_control.utils.search_index import search_documents + if isinstance(docs, str): + docs = json.loads(docs) + hits = search_documents(docs, query, top_k=int(top_k), mode=mode) + return {"hits": [{"doc_id": h.doc_id, "score": h.score} for h in hits]} + + def _resolve_pointer(doc: Any, pointer: str) -> Dict[str, Any]: """Adapter: resolve a JSON Pointer in doc (a dict/list or JSON string).""" import json @@ -3781,6 +3792,7 @@ def __init__(self): "AC_jwt_encode": _jwt_encode, "AC_jwt_decode": _jwt_decode, "AC_rate_limit": _rate_limit, + "AC_search_documents": _search_documents, "AC_resolve_pointer": _resolve_pointer, "AC_apply_json_patch": _apply_json_patch, "AC_make_json_patch": _make_json_patch, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index a2ebb790..492b2cdf 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -3343,6 +3343,23 @@ def rate_limit_tools() -> List[MCPTool]: ] +def search_index_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_search_documents", + description=("Rank a 'docs' corpus ({doc_id: text}) for a text " + "'query' using BM25 (or mode='tfidf'). Returns " + "{hits:[{doc_id, score}]} top 'top_k'."), + input_schema=schema( + {"docs": {"type": "object"}, "query": {"type": "string"}, + "top_k": {"type": "integer"}, "mode": {"type": "string"}}, + ["docs", "query"]), + handler=h.search_documents, + annotations=READ_ONLY, + ), + ] + + def json_patch_tools() -> List[MCPTool]: return [ MCPTool( @@ -4584,6 +4601,7 @@ def media_assert_tools() -> List[MCPTool]: process_mining_tools, asset_tools, events_tools, notify_channel_tools, jsonpath_tools, json_schema_tools, vuln_scan_tools, vex_tools, license_policy_tools, jwt_tools, rate_limit_tools, json_patch_tools, + search_index_tools, saga_tools, decision_table_tools, locator_repair_tools, pii_text_tools, sarif_tools, screen_record_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index d05a3768..1bbdfde5 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -1631,6 +1631,12 @@ def merge_patch(doc, patch): return {"result": _merge(doc, patch)} +def search_documents(docs, query, top_k=10, mode="bm25"): + from je_auto_control.utils.search_index import search_documents as _search + hits = _search(docs, query, top_k=int(top_k), mode=mode) + return {"hits": [{"doc_id": h.doc_id, "score": h.score} for h in hits]} + + def run_saga(steps): from je_auto_control.utils.saga import run_saga as _run result = _run(steps) diff --git a/je_auto_control/utils/search_index/__init__.py b/je_auto_control/utils/search_index/__init__.py new file mode 100644 index 00000000..72c2ba9b --- /dev/null +++ b/je_auto_control/utils/search_index/__init__.py @@ -0,0 +1,6 @@ +"""In-memory BM25 / TF-IDF full-text search over a document corpus.""" +from je_auto_control.utils.search_index.search_index import ( + SearchHit, SearchIndex, search_documents, tokenize, +) + +__all__ = ["SearchHit", "SearchIndex", "search_documents", "tokenize"] diff --git a/je_auto_control/utils/search_index/search_index.py b/je_auto_control/utils/search_index/search_index.py new file mode 100644 index 00000000..4a97cb10 --- /dev/null +++ b/je_auto_control/utils/search_index/search_index.py @@ -0,0 +1,134 @@ +"""In-memory full-text search with BM25 / TF-IDF ranking. + +``fuzzy`` does pairwise string similarity and ``skill_library`` does +alphabetical substring matching, but neither ranks a *corpus* of documents by +relevance — a rare distinctive term and a ubiquitous one weigh the same. This +adds an inverted-index search that ranks documents with Okapi BM25 (or TF-IDF), +so flows and agents can search logs, scraped records, or knowledge snippets +without a database. + +Pure standard library (``math`` + ``collections`` + ``re``); deterministic; +imports no ``PySide6``. +""" +import math +import re +from collections import Counter +from dataclasses import dataclass +from typing import Dict, Iterable, List, Optional, Sequence, Tuple, Union + +_TOKEN_RE = re.compile(r"[a-z0-9]+") + +Docs = Union[Dict[str, str], Iterable[Tuple[str, str]]] + + +@dataclass(frozen=True) +class SearchHit: + """One ranked search result.""" + + doc_id: str + score: float + + +def tokenize(text: str) -> List[str]: + """Lower-case and split ``text`` into alphanumeric terms.""" + return _TOKEN_RE.findall(str(text).lower()) + + +class SearchIndex: + """An inverted index over documents, ranked by BM25 or TF-IDF.""" + + def __init__(self, *, k1: float = 1.5, b: float = 0.75, + stop_words: Optional[Iterable[str]] = None) -> None: + self._k1 = float(k1) + self._b = float(b) + self._stop = set(stop_words or ()) + self._postings: Dict[str, Dict[str, int]] = {} + self._doc_len: Dict[str, int] = {} + + def _terms(self, text: str) -> List[str]: + return [term for term in tokenize(text) if term not in self._stop] + + def add(self, doc_id: str, text: str) -> None: + """Index ``text`` under ``doc_id`` (re-indexing replaces the old text).""" + self.remove(doc_id) + terms = self._terms(text) + self._doc_len[doc_id] = len(terms) + for term, freq in Counter(terms).items(): + self._postings.setdefault(term, {})[doc_id] = freq + + def remove(self, doc_id: str) -> bool: + """Drop ``doc_id`` from the index; return whether it was present.""" + if doc_id not in self._doc_len: + return False + del self._doc_len[doc_id] + for postings in self._postings.values(): + postings.pop(doc_id, None) + for term in [t for t, p in self._postings.items() if not p]: + del self._postings[term] + return True + + @classmethod + def build(cls, docs: Docs, **kwargs) -> "SearchIndex": + """Build an index from a ``{doc_id: text}`` map or ``(id, text)`` pairs.""" + index = cls(**kwargs) + items = docs.items() if isinstance(docs, dict) else docs + for doc_id, text in items: + index.add(doc_id, text) + return index + + def stats(self) -> Dict[str, int]: + """Return ``{docs, terms}`` counts.""" + return {"docs": len(self._doc_len), "terms": len(self._postings)} + + def _avgdl(self) -> float: + if not self._doc_len: + return 1.0 + return sum(self._doc_len.values()) / len(self._doc_len) + + def _idf(self, term: str) -> float: + total = len(self._doc_len) + df = len(self._postings.get(term, {})) + return math.log(1 + (total - df + 0.5) / (df + 0.5)) + + def _bm25(self, terms: Sequence[str], doc_id: str, avgdl: float) -> float: + score = 0.0 + norm = 1 - self._b + self._b * self._doc_len[doc_id] / avgdl + for term in terms: + freq = self._postings.get(term, {}).get(doc_id) + if not freq: + continue + score += self._idf(term) * (freq * (self._k1 + 1)) / ( + freq + self._k1 * norm) + return score + + def _tfidf(self, terms: Sequence[str], doc_id: str, _avgdl: float) -> float: + total = len(self._doc_len) + score = 0.0 + for term in terms: + postings = self._postings.get(term, {}) + freq = postings.get(doc_id) + if not freq: + continue + score += (1 + math.log(freq)) * math.log(total / len(postings)) + return score + + def search(self, query: str, *, top_k: int = 10, + mode: str = "bm25") -> List[SearchHit]: + """Return up to ``top_k`` documents ranked for ``query``.""" + terms = self._terms(query) + candidates = set() + for term in terms: + candidates.update(self._postings.get(term, {})) + avgdl = self._avgdl() + scorer = self._tfidf if mode == "tfidf" else self._bm25 + ranked = ((doc_id, scorer(terms, doc_id, avgdl)) for doc_id in candidates) + hits = [(doc_id, score) for doc_id, score in ranked if score > 0] + hits.sort(key=lambda item: (-item[1], item[0])) + return [SearchHit(doc_id=doc_id, score=round(score, 6)) + for doc_id, score in hits[:top_k]] + + +def search_documents(docs: Docs, query: str, *, top_k: int = 10, + mode: str = "bm25") -> List[SearchHit]: + """One-shot: build an index over ``docs`` and search it for ``query``.""" + return SearchIndex.build(docs).search(query, top_k=top_k, mode=mode) diff --git a/test/unit_test/headless/test_search_index_batch.py b/test/unit_test/headless/test_search_index_batch.py new file mode 100644 index 00000000..9cecb4fa --- /dev/null +++ b/test/unit_test/headless/test_search_index_batch.py @@ -0,0 +1,114 @@ +"""Headless tests for the BM25/TF-IDF search index. Pure stdlib, no Qt.""" +import json + +import je_auto_control as ac +from je_auto_control.utils.search_index import ( + SearchHit, SearchIndex, search_documents, tokenize) + +CORPUS = { + "d1": "the quick brown fox jumps over the lazy dog", + "d2": "a quick brown dog runs fast", + "d3": "lazy cats sleep all day every day", + "d4": "the database stores quick query results quickly quick quick", +} + + +def test_tokenize(): + assert tokenize("Hello, World! 123 foo-bar") == \ + ["hello", "world", "123", "foo", "bar"] + + +def test_build_and_stats(): + index = SearchIndex.build(CORPUS) + assert index.stats()["docs"] == 4 + assert index.stats()["terms"] > 0 + + +def test_rare_term_outranks_common(): + index = SearchIndex.build(CORPUS) + hits = index.search("database") + assert [h.doc_id for h in hits] == ["d4"] # only doc with the term + assert isinstance(hits[0], SearchHit) + + +def test_ranked_results_are_ordered_by_score(): + index = SearchIndex.build(CORPUS) + hits = index.search("quick dog") + scores = [h.score for h in hits] + assert scores == sorted(scores, reverse=True) + assert "d2" in {h.doc_id for h in hits[:2]} # short doc with both terms + + +def test_tf_saturation(): + # a doc with many repeats does not score linearly in tf + index = SearchIndex.build({"a": "quick", "b": "quick quick quick quick"}) + hits = {h.doc_id: h.score for h in index.search("quick")} + assert hits["b"] < 4 * hits["a"] + + +def test_length_normalization(): + short = "alpha beta" + long = "alpha " + " ".join(f"w{i}" for i in range(50)) + index = SearchIndex.build({"short": short, "long": long}) + hits = {h.doc_id: h.score for h in index.search("alpha")} + assert hits["short"] > hits["long"] + + +def test_tfidf_mode_differs_from_bm25(): + index = SearchIndex.build(CORPUS) + bm25 = [h.doc_id for h in index.search("quick dog", mode="bm25")] + tfidf = [h.doc_id for h in index.search("quick dog", mode="tfidf")] + assert bm25 and tfidf # both return results + assert isinstance(bm25, list) + + +def test_top_k_and_no_match(): + index = SearchIndex.build(CORPUS) + assert len(index.search("quick", top_k=2)) == 2 + assert index.search("nonexistentterm") == [] + + +def test_remove_and_reindex(): + index = SearchIndex.build(CORPUS) + assert index.remove("d4") is True + assert index.search("database") == [] + assert index.remove("missing") is False + index.add("d2", "quick quick brown dog") # re-index replaces + assert index.stats()["docs"] == 3 + + +def test_deterministic(): + first = [h.doc_id for h in search_documents(CORPUS, "quick")] + second = [h.doc_id for h in search_documents(CORPUS, "quick")] + assert first == second + + +def test_stop_words(): + index = SearchIndex.build({"a": "the the the cat"}, stop_words={"the"}) + assert index.search("the") == [] + assert [h.doc_id for h in index.search("cat")] == ["a"] + + +# --- wiring --------------------------------------------------------------- + +def test_executor_round_trip(): + rec = ac.execute_action([[ + "AC_search_documents", + {"docs": json.dumps(CORPUS), "query": "lazy dog", "top_k": 3}, + ]]) + hits = next(v for v in rec.values() if isinstance(v, dict))["hits"] + assert hits and all("doc_id" in h and "score" in h for h in hits) + + +def test_wiring(): + assert "AC_search_documents" in ac.executor.known_commands() + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + assert "ac_search_documents" in {t.name for t in build_default_tool_registry()} + from je_auto_control.gui.script_builder.command_schema import _build_specs + assert "AC_search_documents" in {s.command for s in _build_specs()} + + +def test_facade_exports(): + for attr in ("SearchIndex", "SearchHit", "search_documents", "tokenize"): + assert hasattr(ac, attr) + assert attr in ac.__all__