Add BM25/TF-IDF full-text search index

README.md

@@ -13,6 +13,7 @@
 
 ## Table of Contents
 
+- [What's new (2026-06-21) — Full-Text Search (BM25)](#whats-new-2026-06-21--full-text-search-bm25)
 - [What's new (2026-06-21) — JSON Pointer, Patch & Merge Patch](#whats-new-2026-06-21--json-pointer-patch--merge-patch)
 - [What's new (2026-06-21) — Client-Side Rate Limiting](#whats-new-2026-06-21--client-side-rate-limiting)
 - [What's new (2026-06-21) — JSON Web Tokens (JWT)](#whats-new-2026-06-21--json-web-tokens-jwt)
@@ -116,6 +117,12 @@
 
 ---
 
+## What's new (2026-06-21) — Full-Text Search (BM25)
+
+Rank a document corpus by relevance. Full reference: [`docs/source/Eng/doc/new_features/v64_features_doc.rst`](docs/source/Eng/doc/new_features/v64_features_doc.rst).
+
+- **`SearchIndex` / `search_documents` / `tokenize`** (`AC_search_documents`, `ac_search_documents`): `fuzzy` is pairwise and `skill_library` matches substrings alphabetically — neither ranks a corpus by relevance. This adds an inverted-index search ranked with Okapi BM25 (`k1=1.5`, `b=0.75`, `IDF = ln(1+(N−df+0.5)/(df+0.5))`) or TF-IDF, so a rare term out-ranks a common one, term frequency saturates, and long docs are normalized down. Incremental `add`/`remove`, optional stop-words, deterministic ranking. Pure-stdlib `math`+`collections`+`re` — no database.
+
 ## What's new (2026-06-21) — JSON Pointer, Patch & Merge Patch
 
 Address, diff and patch JSON. Full reference: [`docs/source/Eng/doc/new_features/v63_features_doc.rst`](docs/source/Eng/doc/new_features/v63_features_doc.rst).

README/README_zh-CN.md

@@ -12,6 +12,7 @@
 
 ## 目录
 
+- [本次更新 (2026-06-21) — 全文搜索(BM25)](#本次更新-2026-06-21--全文搜索bm25)
 - [本次更新 (2026-06-21) — JSON Pointer、Patch 与 Merge Patch](#本次更新-2026-06-21--json-pointerpatch-与-merge-patch)
 - [本次更新 (2026-06-21) — 客户端速率限制](#本次更新-2026-06-21--客户端速率限制)
 - [本次更新 (2026-06-21) — JSON Web Token(JWT)](#本次更新-2026-06-21--json-web-tokenjwt)
@@ -115,6 +116,12 @@
 
 ---
 
+## 本次更新 (2026-06-21) — 全文搜索(BM25)
+
+依相关性对文档语料排名。完整参考:[`docs/source/Zh/doc/new_features/v64_features_doc.rst`](../docs/source/Zh/doc/new_features/v64_features_doc.rst)。
+
+- **`SearchIndex` / `search_documents` / `tokenize`**(`AC_search_documents`、`ac_search_documents`):`fuzzy` 是成对的,`skill_library` 以字母序匹配子字符串 —— 两者都不会依相关性对语料排名。本功能补上以倒排索引、用 Okapi BM25(`k1=1.5`、`b=0.75`、`IDF = ln(1+(N−df+0.5)/(df+0.5))`)或 TF-IDF 排名的搜索,因此罕见词胜过常见词、词频会饱和、长文档被规范化下调。增量 `add`/`remove`、可选停用词、结果确定。纯标准库 `math`+`collections`+`re` —— 无需数据库。
+
 ## 本次更新 (2026-06-21) — JSON Pointer、Patch 与 Merge Patch
 
 定址、取差异并修补 JSON。完整参考:[`docs/source/Zh/doc/new_features/v63_features_doc.rst`](../docs/source/Zh/doc/new_features/v63_features_doc.rst)。

README/README_zh-TW.md

@@ -12,6 +12,7 @@
 
 ## 目錄
 
+- [本次更新 (2026-06-21) — 全文搜尋(BM25)](#本次更新-2026-06-21--全文搜尋bm25)
 - [本次更新 (2026-06-21) — JSON Pointer、Patch 與 Merge Patch](#本次更新-2026-06-21--json-pointerpatch-與-merge-patch)
 - [本次更新 (2026-06-21) — 用戶端速率限制](#本次更新-2026-06-21--用戶端速率限制)
 - [本次更新 (2026-06-21) — JSON Web Token(JWT)](#本次更新-2026-06-21--json-web-tokenjwt)
@@ -115,6 +116,12 @@
 
 ---
 
+## 本次更新 (2026-06-21) — 全文搜尋(BM25)
+
+依相關性對文件語料排名。完整參考:[`docs/source/Zh/doc/new_features/v64_features_doc.rst`](../docs/source/Zh/doc/new_features/v64_features_doc.rst)。
+
+- **`SearchIndex` / `search_documents` / `tokenize`**(`AC_search_documents`、`ac_search_documents`):`fuzzy` 是成對的,`skill_library` 以字母序比對子字串 —— 兩者都不會依相關性對語料排名。本功能補上以倒排索引、用 Okapi BM25(`k1=1.5`、`b=0.75`、`IDF = ln(1+(N−df+0.5)/(df+0.5))`)或 TF-IDF 排名的搜尋,因此罕見詞勝過常見詞、詞頻會飽和、長文件被正規化下調。增量 `add`/`remove`、可選停用詞、結果具決定性。純標準函式庫 `math`+`collections`+`re` —— 無需資料庫。
+
 ## 本次更新 (2026-06-21) — JSON Pointer、Patch 與 Merge Patch
 
 定址、取差異並修補 JSON。完整參考:[`docs/source/Zh/doc/new_features/v63_features_doc.rst`](../docs/source/Zh/doc/new_features/v63_features_doc.rst)。

docs/source/Eng/doc/new_features/v64_features_doc.rst

@@ -0,0 +1,49 @@
+Full-Text Search (BM25)
+=======================
+
+``fuzzy`` does pairwise string similarity and ``skill_library`` does
+alphabetical substring matching, but neither ranks a *corpus* of documents by
+relevance — a rare distinctive term and a ubiquitous one weigh the same. This
+adds an inverted-index search that ranks documents with Okapi **BM25** (or
+TF-IDF), so flows and agents can search logs, scraped records, or knowledge
+snippets without a database.
+
+Pure standard library (``math`` + ``collections`` + ``re``); deterministic;
+imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import SearchIndex, search_documents
+
+    corpus = {
+        "d1": "the quick brown fox jumps over the lazy dog",
+        "d2": "a quick brown dog runs fast",
+        "d3": "the database stores quick query results",
+    }
+
+    index = SearchIndex.build(corpus)          # or SearchIndex(); index.add(id, text)
+    for hit in index.search("quick dog", top_k=5):
+        print(hit.doc_id, hit.score)
+
+    # one-shot convenience
+    hits = search_documents(corpus, "database", mode="bm25")
+
+``SearchIndex.add`` / ``remove`` keep the index up to date incrementally;
+``build`` indexes a ``{doc_id: text}`` map (or ``(id, text)`` pairs). ``search``
+returns ranked ``SearchHit(doc_id, score)`` results — by default BM25
+(``k1=1.5``, ``b=0.75``), or ``mode="tfidf"``. The scoring is the standard
+Okapi formula with ``IDF = ln(1 + (N − df + 0.5) / (df + 0.5))``, so a rare term
+out-ranks a common one, term-frequency saturates (``k1``), and long documents
+are normalized down (``b``). A ``stop_words`` set can be supplied to drop noise
+terms. Results are deterministic (ties broken by ``doc_id``).
+
+Executor command
+----------------
+
+``AC_search_documents`` takes ``docs`` (a ``{doc_id: text}`` map or JSON
+string), a ``query``, and optional ``top_k`` / ``mode``; it returns
+``{hits: [{doc_id, score}]}``. The same operation is exposed as the MCP tool
+``ac_search_documents`` and as a Script Builder command under **Data**.

docs/source/Eng/eng_index.rst

@@ -86,6 +86,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v61_features_doc
    doc/new_features/v62_features_doc
    doc/new_features/v63_features_doc
+   doc/new_features/v64_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v64_features_doc.rst

@@ -0,0 +1,42 @@
+全文搜尋(BM25)
+===============
+
+``fuzzy`` 做成對的字串相似度,``skill_library`` 做字母序的子字串比對,但兩者都無法依相關性對
+*文件語料*排名 —— 罕見的獨特詞與隨處可見的詞權重相同。本功能補上以倒排索引、用 Okapi **BM25**
+(或 TF-IDF)排名文件的搜尋,讓流程與 agent 不需資料庫即可搜尋日誌、爬取紀錄或知識片段。
+
+純標準函式庫(``math`` + ``collections`` + ``re``);具決定性;不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import SearchIndex, search_documents
+
+    corpus = {
+        "d1": "the quick brown fox jumps over the lazy dog",
+        "d2": "a quick brown dog runs fast",
+        "d3": "the database stores quick query results",
+    }
+
+    index = SearchIndex.build(corpus)          # 或 SearchIndex(); index.add(id, text)
+    for hit in index.search("quick dog", top_k=5):
+        print(hit.doc_id, hit.score)
+
+    # 一次性便利函式
+    hits = search_documents(corpus, "database", mode="bm25")
+
+``SearchIndex.add`` / ``remove`` 以增量方式維護索引;``build`` 索引一個 ``{doc_id: text}`` 對映
+(或 ``(id, text)`` 配對)。``search`` 回傳排名後的 ``SearchHit(doc_id, score)`` 結果 —— 預設
+為 BM25(``k1=1.5``、``b=0.75``),或 ``mode="tfidf"``。評分採標準 Okapi 公式,
+``IDF = ln(1 + (N − df + 0.5) / (df + 0.5))``,因此罕見詞勝過常見詞、詞頻會飽和(``k1``)、長
+文件被正規化下調(``b``)。可提供 ``stop_words`` 集合以濾除雜訊詞。結果具決定性(平手以
+``doc_id`` 決定)。
+
+執行器命令
+----------
+
+``AC_search_documents`` 接受 ``docs``(``{doc_id: text}`` 對映或 JSON 字串)、``query`` 與選用
+的 ``top_k`` / ``mode``,回傳 ``{hits: [{doc_id, score}]}``。同一操作亦以 MCP 工具
+``ac_search_documents`` 以及 Script Builder 中 **Data** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -86,6 +86,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v61_features_doc
    doc/new_features/v62_features_doc
    doc/new_features/v63_features_doc
+   doc/new_features/v64_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -329,6 +329,10 @@
     PatchError, PatchTestFailed, apply_patch, make_merge_patch, make_patch,
     merge_patch, remove_pointer, resolve_pointer, set_pointer,
 )
+# In-memory BM25 / TF-IDF full-text search
+from je_auto_control.utils.search_index import (
+    SearchHit, SearchIndex, search_documents, tokenize,
+)
 # Background popup/interrupt watchdog (unattended automation)
 from je_auto_control.utils.watchdog import (
     PopupWatchdog, WatchdogRule, default_popup_watchdog,
@@ -811,6 +815,7 @@ def start_autocontrol_gui(*args, **kwargs):
     "PatchError", "PatchTestFailed", "apply_patch", "make_merge_patch",
     "make_patch", "merge_patch", "remove_pointer", "resolve_pointer",
     "set_pointer",
+    "SearchHit", "SearchIndex", "search_documents", "tokenize",
     # MCP server
     "AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
     "MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",

je_auto_control/gui/script_builder/command_schema.py

@@ -1155,6 +1155,18 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Validate JSON against a JSON Schema; returns {ok, errors}.",
     ))
+    specs.append(CommandSpec(
+        "AC_search_documents", "Data", "Full-Text Search (BM25)",
+        fields=(
+            FieldSpec("docs", FieldType.STRING,
+                      placeholder='{"d1": "quick brown fox", "d2": "lazy dog"}'),
+            FieldSpec("query", FieldType.STRING, placeholder="quick fox"),
+            FieldSpec("top_k", FieldType.INT, optional=True, default=10),
+            FieldSpec("mode", FieldType.STRING, optional=True,
+                      placeholder="bm25", choices=("bm25", "tfidf")),
+        ),
+        description="Rank a {id: text} corpus for a query; returns {hits}.",
+    ))
     specs.append(CommandSpec(
         "AC_resolve_pointer", "Data", "JSON Pointer: Resolve",
         fields=(

je_auto_control/utils/executor/action_executor.py

@@ -2928,6 +2928,17 @@ def _rate_limit(name: str, rate: float = 1.0, capacity: float = 1.0,
             "wait": round(bucket.time_until_available(float(n)), 4)}
 
 
+def _search_documents(docs: Any, query: str, top_k: int = 10,
+                      mode: str = "bm25") -> Dict[str, Any]:
+    """Adapter: BM25/TF-IDF search a {doc_id: text} corpus (dict or JSON str)."""
+    import json
+    from je_auto_control.utils.search_index import search_documents
+    if isinstance(docs, str):
+        docs = json.loads(docs)
+    hits = search_documents(docs, query, top_k=int(top_k), mode=mode)
+    return {"hits": [{"doc_id": h.doc_id, "score": h.score} for h in hits]}
+
+
 def _resolve_pointer(doc: Any, pointer: str) -> Dict[str, Any]:
     """Adapter: resolve a JSON Pointer in doc (a dict/list or JSON string)."""
     import json
@@ -3781,6 +3792,7 @@ def __init__(self):
             "AC_jwt_encode": _jwt_encode,
             "AC_jwt_decode": _jwt_decode,
             "AC_rate_limit": _rate_limit,
+            "AC_search_documents": _search_documents,
             "AC_resolve_pointer": _resolve_pointer,
             "AC_apply_json_patch": _apply_json_patch,
             "AC_make_json_patch": _make_json_patch,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -3343,6 +3343,23 @@ def rate_limit_tools() -> List[MCPTool]:
     ]
 
 
+def search_index_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_search_documents",
+            description=("Rank a 'docs' corpus ({doc_id: text}) for a text "
+                         "'query' using BM25 (or mode='tfidf'). Returns "
+                         "{hits:[{doc_id, score}]} top 'top_k'."),
+            input_schema=schema(
+                {"docs": {"type": "object"}, "query": {"type": "string"},
+                 "top_k": {"type": "integer"}, "mode": {"type": "string"}},
+                ["docs", "query"]),
+            handler=h.search_documents,
+            annotations=READ_ONLY,
+        ),
+    ]
+
+
 def json_patch_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -4584,6 +4601,7 @@ def media_assert_tools() -> List[MCPTool]:
     process_mining_tools, asset_tools, events_tools, notify_channel_tools,
     jsonpath_tools, json_schema_tools, vuln_scan_tools, vex_tools,
     license_policy_tools, jwt_tools, rate_limit_tools, json_patch_tools,
+    search_index_tools,
     saga_tools, decision_table_tools, locator_repair_tools,
     pii_text_tools, sarif_tools,
     screen_record_tools,

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -1631,6 +1631,12 @@ def merge_patch(doc, patch):
     return {"result": _merge(doc, patch)}
 
 
+def search_documents(docs, query, top_k=10, mode="bm25"):
+    from je_auto_control.utils.search_index import search_documents as _search
+    hits = _search(docs, query, top_k=int(top_k), mode=mode)
+    return {"hits": [{"doc_id": h.doc_id, "score": h.score} for h in hits]}
+
+
 def run_saga(steps):
     from je_auto_control.utils.saga import run_saga as _run
     result = _run(steps)

je_auto_control/utils/search_index/__init__.py

@@ -0,0 +1,6 @@
+"""In-memory BM25 / TF-IDF full-text search over a document corpus."""
+from je_auto_control.utils.search_index.search_index import (
+    SearchHit, SearchIndex, search_documents, tokenize,
+)
+
+__all__ = ["SearchHit", "SearchIndex", "search_documents", "tokenize"]