Opt/plan

[augurier]

Task Description

This PR covers three features for OceanBase SeekDB's full-text search and document processing capabilities:

• task2: Document AI Functions — Introduce LOAD_FILE() and AI_SPLIT_DOCUMENT() SQL functions that enable loading external files and splitting documents into structured chunks directly within SQL queries, extending OceanBase's AI integration surface.
• task3: IK Custom Dictionary — Add a FULLTEXT_DICT system table and ALTER SYSTEM REFRESH FULLTEXT DICT command to support runtime-loading of user-defined IK dictionaries without restarting services, enabling more flexible Chinese text segmentation.
• task4: FTS Tokenization Hot-Path Optimization — Refactor the internal token representation from ObFTWord to ObFTToken, restructure the plugin helper layer, and migrate the dictionary subsystem data model, aiming to reduce allocation overhead and improve tokenization throughput on hot paths.

Solution Description

• Document AI Functions (ob_expr_load_file.cpp, ob_ai_split_document.cpp): Implement LOAD_FILE(path) to read file contents into a string column, and AI_SPLIT_DOCUMENT(content, ...) to chunk documents via configurable strategies. Both are wired into the SQL expression evaluator and DML resolver.
• IK Custom Dictionary (ob_ft_dict_hub.cpp, ob_alter_system_executor.cpp): Introduce FULLTEXT_DICT schema for storing user-defined dictionary entries, backed by a table iterator (ob_ft_dict_table_iter). ALTER SYSTEM REFRESH FULLTEXT DICT triggers a dictionary reload into the in-memory trie, making new terms available for IK segmentation immediately.
• FTS Hot-Path Optimization: Replace ObFTWord with a lightweight ObFTToken struct, restructure ObFTsPluginHelper to separate parser lifecycle from per-token processing, and migrate dictionary data model (ob_ft_dict_def, ob_ft_range_dict, ob_ft_dat_dict) for more efficient memory layout and reduced copy overhead during tokenization.

Passed Regressions

• Unit test added: test_ob_ai_split_document.cpp covering AI_SPLIT_DOCUMENT core logic.
• Manual testing performed for LOAD_FILE, ALTER SYSTEM REFRESH FULLTEXT DICT, and FTS query correctness with IK parser.
• Existing full-text search regression suites pass (IK / Ngram / Whitespace parsers).

Upgrade Compatibility

• Backward compatible: No DDL changes to existing user tables. FULLTEXT_DICT is a new system table; existing indexes are unaffected.
• Upgrade path: After upgrading, run ALTER SYSTEM REFRESH FULLTEXT DICT to load any custom dictionary entries if previously configured via system variables.
• ObFTToken change is internal only; no wire-protocol or SQL-level incompatibilities.

Release Note

• team: 来桂林吃kfc

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 100.00 / 100
ik_custom_dict: 100 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 91.16 / 100
mean_improvement: 45.58%
full_score_improvement: 50.00%

build_improvement: 29.19%
  build_ik_all_sec: baseline=35.2836, current=24.686, improvement=30.04%
  build_ik_content_sec: baseline=28.3764, current=20.033, improvement=29.40%
  build_beng_en_sec: baseline=14.7578, current=10.605, improvement=28.14%
tokenize_improvement: 36.47%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.371, improvement=51.49%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.332, improvement=21.44%
query_improvement: 71.08%
  query_cn_avg_ms: baseline=16.6628, current=5.6801, improvement=65.91%
  query_beng_avg_ms: baseline=24.3042, current=8.6708, improvement=64.32%
  query_mixed_avg_ms: baseline=17.5593, current=6.6312, improvement=62.24%
  query_limit_avg_ms: baseline=16.2334, current=1.3244, improvement=91.84%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-18 06:30:43
label:                  vldb-ci-29633235600-1
git_head:               eac578a
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2707
select1_stdev_ms:       0.0020
raw_load_sec:           1.490
raw_load_rows_per_sec:  13422.8
build_ik_all_sec:       24.686
build_ik_content_sec:   20.033
build_beng_en_sec:      10.605
build_total_sec:        55.337
----------------------------------------
tokenize_ik_avg_ms:     0.3710
tokenize_ik_median_ms:  0.3717
tokenize_ik_stdev_ms:   0.0046
tokenize_beng_avg_ms:   0.3320
tokenize_beng_median_ms:0.3330
tokenize_beng_stdev_ms: 0.0029
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        5.6801
query_cn_stdev_ms:      0.0056
query_beng_hits:        11000
query_beng_avg_ms:      8.6708
query_beng_stdev_ms:    0.0084
query_mixed_hits:       7332
query_mixed_avg_ms:     6.6312
query_mixed_stdev_ms:   0.0033
query_limit_hits:       20
query_limit_avg_ms:     1.3244
query_limit_stdev_ms:   0.0039
========================================

Workflow run