IMP: add document AI, custom IK dictionaries, and optimize FTS"

[WeiJinMei0]

Task Description

This pull request implements the second, third, and fourth VLDB 2026 summer school tasks for SeekDB.

1. Document AI functions

   • Add LOAD_FILE(location_name, file_name) to securely read a file registered through a SeekDB LOCATION and return its contents as a BLOB.
   • Add the AI_SPLIT_DOCUMENT(content[, parameters]) table function to split plain-text or Markdown documents by word or sentence and return chunk ID, offset, length, and text.

2. Custom dictionaries for the IK full-text parser

   • Allow users to create validated FULLTEXT_DICT tables and associate custom main, stopword, and quantifier dictionaries with an IK full-text index.
   • Add ALTER SYSTEM REFRESH FULLTEXT DICT so dictionary changes can be made visible to subsequent tokenization and indexing operations.

3. Full-text index build performance

   • Optimize the tokenizer, sorting framework, DDL pipeline, sort-key representation, position-list storage, and DAG observability used by large FTS index builds.
   • Keep post-build FTS execution on the established row-by-row path for now because the new vectorized table-scan output path is not yet ready for production use.

Solution Description

Document AI

• Implement LOAD_FILE as a SQL expression with argument validation and BLOB result typing.
• Resolve the named LOCATION through the schema guard, verify the session's read privilege, build the target URI from the location and file name, and read the file through the storage I/O adapter.
• Validate file size and actual bytes read, support NULL propagation, and reject missing locations, inaccessible locations, invalid paths, and files larger than the supported LOB limit.
• Implement AI_SPLIT_DOCUMENT as a function-table expression with four output columns: CHUNK_ID, CHUNK_OFFSET, CHUNK_LENGTH, and CHUNK_TEXT.
• Parse the optional JSON parameters and support:
  • type: text or markdown;
  • by: word or sentence;
  • max: maximum units per chunk;
  • overlap: units shared by adjacent chunks.
• Preserve Markdown section headings in generated chunks and calculate offsets and lengths for each returned row.

IK custom dictionary

• Extend CREATE TABLE parsing and schema validation for FULLTEXT_DICT='Y' dictionary tables.
• Require an index-organized UTF-8 (utf8mb4) table containing exactly one VARCHAR(1..500) primary-key column named word.
• Extend FTS parser properties with dict_table, stopword_table, quantifier_table, and ik_mode, and pass the selected dictionaries into IK parser initialization.
• Load tenant dictionary-table contents into the FTS dictionary cache and use them for tokenization and full-text index construction. Custom main dictionaries intentionally replace the built-in main dictionary for that parser configuration.
• Implement ALTER SYSTEM REFRESH FULLTEXT DICT [database.]table through the SQL parser, resolver, command executor, and dictionary hub. Refresh invalidates the selected custom dictionary cache so that the next use reloads current table contents.
• Support dictionary use from both FTS index PARSER_PROPERTIES and the TOKENIZE() function.
• Enforce schema and DDL restrictions that protect dictionary tables while they are referenced by full-text indexes.

Full-text index optimization

• Reduce tokenizer hot-path allocation and dispatch overhead through reusable parser state, reusable segmented containers, separated metadata/scratch allocation, combined character classification, cached charset callbacks, and reusable IK arbitration state.
• Consolidate token filtering, normalization, stopword checks, and grouping in the FTS token-processing pipeline, with cached dictionary lookup for repeated operations.
• Refactor sorting into reusable strategy, resource-management, chunk-building, external-merge, and row-store components shared by SQL and storage paths.
• Add encoded order-preserving sort keys and fixed-key/radix-sort support where applicable, while retaining comparison-sort fallbacks for unsupported layouts.
• Add storage-vector sorting and sort-key/payload separation to reduce materialization and I/O during merge processing.
• Rework FTS DDL construction around partition-local and multi-stage DAG pipelines for sampling, doc-word output, merge preparation, word-doc output, and final merge.
• Add persisted FTS sampling boundaries, reusable DDL sort providers, parallel self-suspending merge tasks, and macro-block output pipelines.
• Add compact FTS position-list encoding and the generated position-list expression required by the extended auxiliary-table layout.
• Add FTS-aware granule allocation, parallel DDL range handling, statistics collection, and the __all_virtual_ddl_dag_monitor observability path.
• Temporarily set the FTS DDL table scan max_batch_size_ to 0, remove the unfinished dedicated FTS batch-output code, and resolve generated FTS expressions by operator type on the stable row path. This fallback should be removed after post-build FTS vectorization is fully supported and validated.

Passed Regressions

The following focused mysqltest cases and expected-result files are included:

• tools/deploy/mysql_test/test_suite/ai_funcs/t/load_file.test
  • Creates a temporary local file and LOCATION.
  • Verifies returned file content and BLOB byte length.
• tools/deploy/mysql_test/test_suite/ai_funcs/t/ai_split_document.test
  • Verifies sentence splitting, word-window overlap, chunk metadata, and Markdown heading preservation.
• tools/deploy/mysql_test/test_suite/ai_funcs/t/ik_custom_dict.test
  • Verifies dictionary-table creation, IK index integration, custom word matching, replacement semantics, dynamic dictionary updates, cache refresh, and matching of newly indexed data.
• Existing FTS parser-property unit coverage verifies serialization and retrieval of custom main, stopword, and quantifier dictionary properties.
• The large FTS benchmark is available at tools/benchmark/fts_large_bench.sh and is also intended to run through GitHub Actions for scoring.

Verification status when this PR description was prepared: the test and expected-result files were inspected, but the mysqltest suites and large FTS benchmark were not executed locally. The final GitHub Actions regression and benchmark links/results should be attached to the PR after completion; only the latest commit result is used for scoring.

No existing test file was modified solely to bypass or weaken validation.

Upgrade Compatibility

The new SQL features are additive and do not require a manual data migration.

• Existing SQL statements, tables, indexes, and FTS indexes that do not use these features retain their previous behavior.
• LOAD_FILE and AI_SPLIT_DOCUMENT introduce new function interfaces without changing existing function contracts.
• FULLTEXT_DICT, its parser properties, and the refresh command are opt-in. Existing IK indexes continue to use built-in dictionaries unless custom dictionary properties are supplied.
• Dictionary refresh affects tokenization and rows indexed after the refresh. Existing FTS index entries are not automatically re-tokenized; users must rebuild the full-text index if historical rows need to use the updated dictionary.
• FTS build optimizations preserve the logical auxiliary-table and query behavior. The temporary row-mode fallback changes only the internal construction strategy and requires no upgrade action.
• Position-list/auxiliary-table format changes must remain version-aware when mixed-version upgrade support is required; rolling-upgrade and downgrade coverage should be confirmed by the release pipeline before backporting this work to an older release line.

Other Information

• Target branch: vldb_2026 in oceanbase/seekdb.
• The benchmark score is based on tools/benchmark/fts_large_bench.sh; GitHub Actions results from the last commit are authoritative for the competition.
• Review focus for Document AI should include LOCATION privilege enforcement, path handling, LOB limits, invalid JSON parameters, chunk boundary/overlap behavior, and Markdown offsets.
• Review focus for custom dictionaries should include schema validation, tenant/database qualification, cache invalidation, concurrent refresh/use, DDL dependency restrictions, and the distinction between refreshing future tokenization and rebuilding existing index data.
• Review focus for FTS performance should include memory ownership, spill/merge correctness, DAG retry and recovery behavior, range-boundary persistence, auxiliary-table ordering, position-list compatibility, and correctness under the temporary non-vectorized fallback.
• The forced FTS row-mode fallback is intentional and marked with a TODO. It trades some potential index-build throughput for the correctness and stability of the established execution path until vectorized post-build FTS output is ready.

Release Note

Adds Document AI file loading and document splitting functions, introduces refreshable custom dictionaries for the IK full-text parser, and improves the performance and observability of full-text index construction while retaining a stable row-based fallback for post-build FTS processing.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ WeiJinMei0
❌ SeekDB Developer

---

SeekDB Developer seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 100.00 / 100
ik_custom_dict: 100 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 76.41 / 100
mean_improvement: 38.21%
full_score_improvement: 50.00%

build_improvement: 28.33%
  build_ik_all_sec: baseline=35.2836, current=26.078, improvement=26.09%
  build_ik_content_sec: baseline=28.3764, current=19.55, improvement=31.10%
  build_beng_en_sec: baseline=14.7578, current=10.655, improvement=27.80%
tokenize_improvement: 37.93%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.3629, improvement=52.55%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.3241, improvement=23.31%
query_improvement: 48.36%
  query_cn_avg_ms: baseline=16.6628, current=7.7297, improvement=53.61%
  query_beng_avg_ms: baseline=24.3042, current=10.8632, improvement=55.30%
  query_mixed_avg_ms: baseline=17.5593, current=8.4413, improvement=51.93%
  query_limit_avg_ms: baseline=16.2334, current=10.9444, improvement=32.58%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-18 07:49:57
label:                  vldb-ci-29634944375-1
git_head:               b6f896d
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2656
select1_stdev_ms:       0.0013
raw_load_sec:           1.507
raw_load_rows_per_sec:  13271.4
build_ik_all_sec:       26.078
build_ik_content_sec:   19.550
build_beng_en_sec:      10.655
build_total_sec:        56.296
----------------------------------------
tokenize_ik_avg_ms:     0.3629
tokenize_ik_median_ms:  0.3631
tokenize_ik_stdev_ms:   0.0012
tokenize_beng_avg_ms:   0.3241
tokenize_beng_median_ms:0.3242
tokenize_beng_stdev_ms: 0.0018
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        7.7297
query_cn_stdev_ms:      0.0228
query_beng_hits:        11000
query_beng_avg_ms:      10.8632
query_beng_stdev_ms:    0.0038
query_mixed_hits:       7332
query_mixed_avg_ms:     8.4413
query_mixed_stdev_ms:   0.0342
query_limit_hits:       20
query_limit_avg_ms:     10.9444
query_limit_stdev_ms:   0.0032
========================================

Workflow run