seekdb_v1#1129
Open
shiyunchengAI wants to merge 1 commit into
Open
Conversation
|
|
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 100.00 / 100 ik_custom_dict: 100 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 77.98 / 100 mean_improvement: 38.99% full_score_improvement: 50.00% build_improvement: 30.54% build_ik_all_sec: baseline=35.2836, current=24.194, improvement=31.43% build_ik_content_sec: baseline=28.3764, current=19.973, improvement=29.61% build_beng_en_sec: baseline=14.7578, current=10.244, improvement=30.59% tokenize_improvement: 37.29% tokenize_ik_avg_ms: baseline=0.76478, current=0.3671, improvement=52.00% tokenize_beng_avg_ms: baseline=0.42262, current=0.3272, improvement=22.58% query_improvement: 49.14% query_cn_avg_ms: baseline=16.6628, current=7.5886, improvement=54.46% query_beng_avg_ms: baseline=24.3042, current=10.5753, improvement=56.49% query_mixed_avg_ms: baseline=17.5593, current=8.269, improvement=52.91% query_limit_avg_ms: baseline=16.2334, current=10.925, improvement=32.70% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-18 04:38:04 label: vldb-ci-29630058110-1 git_head: f4a8b48 git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2871 select1_stdev_ms: 0.0133 raw_load_sec: 1.500 raw_load_rows_per_sec: 13333.3 build_ik_all_sec: 24.194 build_ik_content_sec: 19.973 build_beng_en_sec: 10.244 build_total_sec: 54.425 ---------------------------------------- tokenize_ik_avg_ms: 0.3671 tokenize_ik_median_ms: 0.3596 tokenize_ik_stdev_ms: 0.0112 tokenize_beng_avg_ms: 0.3272 tokenize_beng_median_ms:0.3271 tokenize_beng_stdev_ms: 0.0005 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 7.5886 query_cn_stdev_ms: 0.0036 query_beng_hits: 11000 query_beng_avg_ms: 10.5753 query_beng_stdev_ms: 0.0147 query_mixed_hits: 7332 query_mixed_avg_ms: 8.2690 query_mixed_stdev_ms: 0.0258 query_limit_hits: 20 query_limit_avg_ms: 10.9250 query_limit_stdev_ms: 0.0609 ======================================== |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task Description
This pull request implements the second, third, and fourth VLDB 2026 summer school tasks for SeekDB.
Document AI functions
Add LOAD_FILE(location_name, file_name) to securely read a file registered through a SeekDB LOCATION and return its contents as a BLOB.
Add the AI_SPLIT_DOCUMENT(content[, parameters]) table function to split plain-text or Markdown documents by word or sentence and return chunk ID, offset, length, and text.
Custom dictionaries for the IK full-text parser
Allow users to create validated FULLTEXT_DICT tables and associate custom main, stopword, and quantifier dictionaries with an IK full-text index.
Add ALTER SYSTEM REFRESH FULLTEXT DICT so dictionary changes can be made visible to subsequent tokenization and indexing operations.
Full-text index build performance
Optimize the tokenizer, sorting framework, DDL pipeline, sort-key representation, position-list storage, and DAG observability used by large FTS index builds.
Keep post-build FTS execution on the established row-by-row path for now because the new vectorized table-scan output path is not yet ready for production use.
Solution Description
Document AI
Implement LOAD_FILE as a SQL expression with argument validation and BLOB result typing.
Resolve the named LOCATION through the schema guard, verify the session's read privilege, build the target URI from the location and file name, and read the file through the storage I/O adapter.
Validate file size and actual bytes read, support NULL propagation, and reject missing locations, inaccessible locations, invalid paths, and files larger than the supported LOB limit.
Implement AI_SPLIT_DOCUMENT as a function-table expression with four output columns: CHUNK_ID, CHUNK_OFFSET, CHUNK_LENGTH, and CHUNK_TEXT.
Parse the optional JSON parameters and support:
type: text or markdown;
by: word or sentence;
max: maximum units per chunk;
overlap: units shared by adjacent chunks.
Preserve Markdown section headings in generated chunks and calculate offsets and lengths for each returned row.
IK custom dictionary
Extend CREATE TABLE parsing and schema validation for FULLTEXT_DICT='Y' dictionary tables.
Require an index-organized UTF-8 (utf8mb4) table containing exactly one VARCHAR(1..500) primary-key column named word.
Extend FTS parser properties with dict_table, stopword_table, quantifier_table, and ik_mode, and pass the selected dictionaries into IK parser initialization.
Load tenant dictionary-table contents into the FTS dictionary cache and use them for tokenization and full-text index construction. Custom main dictionaries intentionally replace the built-in main dictionary for that parser configuration.
Implement ALTER SYSTEM REFRESH FULLTEXT DICT [database.]table through the SQL parser, resolver, command executor, and dictionary hub. Refresh invalidates the selected custom dictionary cache so that the next use reloads current table contents.
Support dictionary use from both FTS index PARSER_PROPERTIES and the TOKENIZE() function.
Enforce schema and DDL restrictions that protect dictionary tables while they are referenced by full-text indexes.
Full-text index optimization
Reduce tokenizer hot-path allocation and dispatch overhead through reusable parser state, reusable segmented containers, separated metadata/scratch allocation, combined character classification, cached charset callbacks, and reusable IK arbitration state.
Consolidate token filtering, normalization, stopword checks, and grouping in the FTS token-processing pipeline, with cached dictionary lookup for repeated operations.
Refactor sorting into reusable strategy, resource-management, chunk-building, external-merge, and row-store components shared by SQL and storage paths.
Add encoded order-preserving sort keys and fixed-key/radix-sort support where applicable, while retaining comparison-sort fallbacks for unsupported layouts.
Add storage-vector sorting and sort-key/payload separation to reduce materialization and I/O during merge processing.
Rework FTS DDL construction around partition-local and multi-stage DAG pipelines for sampling, doc-word output, merge preparation, word-doc output, and final merge.
Add persisted FTS sampling boundaries, reusable DDL sort providers, parallel self-suspending merge tasks, and macro-block output pipelines.
Add compact FTS position-list encoding and the generated position-list expression required by the extended auxiliary-table layout.
Add FTS-aware granule allocation, parallel DDL range handling, statistics collection, and the _all_virtual_ddl_dag_monitor observability path.
Temporarily set the FTS DDL table scan max_batch_size to 0, remove the unfinished dedicated FTS batch-output code, and resolve generated FTS expressions by operator type on the stable row path. This fallback should be removed after post-build FTS vectorization is fully supported and validated.
Passed Regressions
The following focused mysqltest cases and expected-result files are included:
tools/deploy/mysql_test/test_suite/ai_funcs/t/load_file.test
Creates a temporary local file and LOCATION.
Verifies returned file content and BLOB byte length.
tools/deploy/mysql_test/test_suite/ai_funcs/t/ai_split_document.test
Verifies sentence splitting, word-window overlap, chunk metadata, and Markdown heading preservation.
tools/deploy/mysql_test/test_suite/ai_funcs/t/ik_custom_dict.test
Verifies dictionary-table creation, IK index integration, custom word matching, replacement semantics, dynamic dictionary updates, cache refresh, and matching of newly indexed data.
Existing FTS parser-property unit coverage verifies serialization and retrieval of custom main, stopword, and quantifier dictionary properties.
The large FTS benchmark is available at tools/benchmark/fts_large_bench.sh and is also intended to run through GitHub Actions for scoring.
Verification status when this PR description was prepared: the test and expected-result files were inspected, but the mysqltest suites and large FTS benchmark were not executed locally. The final GitHub Actions regression and benchmark links/results should be attached to the PR after completion; only the latest commit result is used for scoring.
No existing test file was modified solely to bypass or weaken validation.
Upgrade Compatibility
The new SQL features are additive and do not require a manual data migration.
Existing SQL statements, tables, indexes, and FTS indexes that do not use these features retain their previous behavior.
LOAD_FILE and AI_SPLIT_DOCUMENT introduce new function interfaces without changing existing function contracts.
FULLTEXT_DICT, its parser properties, and the refresh command are opt-in. Existing IK indexes continue to use built-in dictionaries unless custom dictionary properties are supplied.
Dictionary refresh affects tokenization and rows indexed after the refresh. Existing FTS index entries are not automatically re-tokenized; users must rebuild the full-text index if historical rows need to use the updated dictionary.
FTS build optimizations preserve the logical auxiliary-table and query behavior. The temporary row-mode fallback changes only the internal construction strategy and requires no upgrade action.
Position-list/auxiliary-table format changes must remain version-aware when mixed-version upgrade support is required; rolling-upgrade and downgrade coverage should be confirmed by the release pipeline before backporting this work to an older release line.
Other Information
Target branch: vldb_2026 in oceanbase/seekdb.
The benchmark score is based on tools/benchmark/fts_large_bench.sh; GitHub Actions results from the last commit are authoritative for the competition.
Review focus for Document AI should include LOCATION privilege enforcement, path handling, LOB limits, invalid JSON parameters, chunk boundary/overlap behavior, and Markdown offsets.
Review focus for custom dictionaries should include schema validation, tenant/database qualification, cache invalidation, concurrent refresh/use, DDL dependency restrictions, and the distinction between refreshing future tokenization and rebuilding existing index data.
Review focus for FTS performance should include memory ownership, spill/merge correctness, DAG retry and recovery behavior, range-boundary persistence, auxiliary-table ordering, position-list compatibility, and correctness under the temporary non-vectorized fallback.
The forced FTS row-mode fallback is intentional and marked with a TODO. It trades some potential index-build throughput for the correctness and stability of the established execution path until vectorized post-build FTS output is ready.
Release Note
Adds Document AI file loading and document splitting functions, introduces refreshable custom dictionaries for the IK full-text parser, and improves the performance and observability of full-text index construction while retaining a stable row-based fallback for post-build FTS processing.