feat: Implement LOAD_FILE and AI_SPLIT_DOCUMENT functions

[guojunyuan2004]

Add two Document AI functions callable from SQL, per docs/2_DocumentAIFunctions.md:

• LOAD_FILE(location_name, file_name): read a file from a registered file:// LOCATION and return its contents as a BLOB.

• AI_SPLIT_DOCUMENT(content, parameters): a function table that splits text or markdown content into chunks, returning one row per chunk with CHUNK_ID / CHUNK_OFFSET / CHUNK_LENGTH / CHUNK_TEXT. Parameters control type (text|markdown), by (word|sentence), max chunk size, and overlap between adjacent chunks.

Task Description

Implements task 2 of docs/2_DocumentAIFunctions.md (Document AI Functions). The database currently can only store and retrieve unstructured document content; this PR brings AI-capable functions into the SQL execution path so users can ingest local files and split documents into chunks directly from SQL.

Two functions are added:

1. LOAD_FILE — import a local file into seekdb as a BLOB.
2. AI_SPLIT_DOCUMENT — split text/markdown content into chunk rows for downstream processing.

Solution Description

Parser & resolver

• Register two new item types T_FUN_SYS_LOAD_FILE and T_FUN_SYS_AI_SPLIT_DOCUMENT, the N_ macro entries, and a non-reserved keyword.
• Parse FROM AI_SPLIT_DOCUMENT(...) as a function table in the MySQL-mode grammar (sql_parser_mysql_mode.y).
• In the DML resolver, resolve AI_SPLIT_DOCUMENT into a function table and create its 4 output columns (chunk_id / chunk_offset / chunk_length / chunk_text) with the correct namespace paths.

LOAD_FILE (scalar expr)

• Looks up the named LOCATION via the schema guard, enforces that the caller has location-read privilege, and restricts access to file:// locations only.
• Builds the file path from the LOCATION URL + file name with path-safety checks: rejects absolute paths, .. traversal components, and embedded NUL bytes.
• Reads the file via ObFileReader and returns the bytes as a BLOB.

AI_SPLIT_DOCUMENT (function-table expr)

• An ObExprOperatorCtx materializes all chunks on first eval, then returns one chunk per call (OB_ITER_END when exhausted). The ObFunctionTableOp reads the 4 column values from the rt_ctx; multi-column dispatch is guarded on item type.
• Splitting operates on byte offsets into the original content; chunk text is deep-copied into the rt_ctx arena.
• Parameters parsed from the JSON second arg: type (text|markdown, default markdown), by (word|sentence, default word), max (units per chunk, default 256), overlap (units shared with the previous chunk, default 0).
• Sentence boundaries: ASCII . ! ? and CJK fullwidth 。 ！ ？, recognized only when followed by whitespace or EOT, so tokens like 3.14, Mr., e.g. are not mis-split.
• Markdown mode: each chunk carries the heading of the section it belongs to.

Passed Regressions

Added two mysqltest cases under tools/deploy/mysql_test/test_suite/ai_funcs/t/:

• load_file.test — creates a scratch file under $MYSQL_TMP_DIR, registers a file:// LOCATION, and asserts load_file(...) returns the full content and that length(load_file(...)) matches.
• ai_split_document.test — covers (a) one sentence per chunk, (b) word split with a sliding window (max=3, overlap=1), and (c) markdown split where each chunk carries its section heading.

All cases match the expected output documented in docs/2_DocumentAIFunctions.md.

Upgrade Compatibility

Backward compatible. Both functions are purely additive:

• New item-type enum values and a new non-reserved keyword are appended; no existing values are renumbered.
• LOAD_FILE and AI_SPLIT_DOCUMENT are opt-in — no behavior changes for existing queries.
• LOAD_FILE requires an existing CREATE LOCATION and location-read privilege; it only supports file:// locations, matching seekdb's current external-storage scope.

Other Information

• Squashed from the ai_split work on vldb_2026 (6 commits) into a single commit on feat/document_ai_functions, based on a640cc32.
• Branch: feat/document_ai_functions → targets develop.

Release Note

Added two Document AI SQL functions:

• LOAD_FILE(location_name, file_name) — read a file from a registered file:// LOCATION and return it as a BLOB.
• AI_SPLIT_DOCUMENT(content, parameters) — a table function that splits text or markdown into chunk rows (chunk_id, chunk_offset, chunk_length, chunk_text), with configurable type, by, max, and overlap parameters.

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 0.00 / 100
ik_custom_dict: 0 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 2.34 / 100
mean_improvement: 1.17%
full_score_improvement: 50.00%

build_improvement: 0.55%
  build_ik_all_sec: baseline=35.2836, current=34.908, improvement=1.06%
  build_ik_content_sec: baseline=28.3764, current=28.182, improvement=0.69%
  build_beng_en_sec: baseline=14.7578, current=14.774, improvement=-0.11%
tokenize_improvement: 0.79%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.7567, improvement=1.06%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.4204, improvement=0.53%
query_improvement: 2.17%
  query_cn_avg_ms: baseline=16.6628, current=16.2553, improvement=2.45%
  query_beng_avg_ms: baseline=24.3042, current=23.7008, improvement=2.48%
  query_mixed_avg_ms: baseline=17.5593, current=17.1258, improvement=2.47%
  query_limit_avg_ms: baseline=16.2334, current=16.0282, improvement=1.26%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-18 13:38:32
label:                  vldb-ci-29645903604-1
git_head:               9dcc2fe
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2055
select1_stdev_ms:       0.0043
raw_load_sec:           1.526
raw_load_rows_per_sec:  13106.2
build_ik_all_sec:       34.908
build_ik_content_sec:   28.182
build_beng_en_sec:      14.774
build_total_sec:        77.881
----------------------------------------
tokenize_ik_avg_ms:     0.7567
tokenize_ik_median_ms:  0.7534
tokenize_ik_stdev_ms:   0.0103
tokenize_beng_avg_ms:   0.4204
tokenize_beng_median_ms:0.4165
tokenize_beng_stdev_ms: 0.0075
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        16.2553
query_cn_stdev_ms:      0.0155
query_beng_hits:        11000
query_beng_avg_ms:      23.7008
query_beng_stdev_ms:    0.0167
query_mixed_hits:       7332
query_mixed_avg_ms:     17.1258
query_mixed_stdev_ms:   0.0134
query_limit_hits:       20
query_limit_avg_ms:     16.0282
query_limit_stdev_ms:   0.0059
========================================

Workflow run