My branch-test#1137
Open
shenriri wants to merge 8 commits into
Open
Conversation
Implement the two Document AI functions for the ai_funcs test suite: LOAD_FILE(location_name, file_name) -> BLOB - new scalar sys expr ObExprLoadFile (T_FUN_SYS_LOAD_FILE): looks up the LOCATION schema by name, joins its URL with the file name and reads the file through ObBackupIoAdapter, returning the content as a binary lob AI_SPLIT_DOCUMENT(content [, parameters]) table function - new non-reserved keyword + grammar rule producing T_AI_SPLIT_DOCUMENT_EXPRESSION, resolved as a JSON_TABLE-family table item (OB_AI_SPLIT_DOC_TABLE_TYPE) with four fixed output columns chunk_id / chunk_offset / chunk_length / chunk_text - AiSplitDocTableFunc splits the document per parameters json: type text|markdown (default markdown), by word|sentence (default word), max units per chunk (default 256), overlap for sliding windows; markdown mode sections the text at '#' heading lines and prefixes each chunk with its section heading - spec serialization, printer (view expansion) and rescan support Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implement the Custom Dictionary feature for the IK fulltext parser: - CREATE TABLE ... FULLTEXT_DICT='Y' table option: new FULLTEXT_DICT non-reserved keyword and table option, accepted for CREATE TABLE (value must be 'Y'/'N', rejected in ALTER TABLE like ORGANIZATION). - ALTER SYSTEM REFRESH FULLTEXT DICT [db.]table statement: new DICT keyword, T_REFRESH_FULLTEXT_DICT parse node, resolver, stmt and executor wired through the standard system-command chain. Accepts the unquoted db.table / table forms and the double-quoted string forms. Custom dictionaries are re-read from the dictionary table on every parser instantiation, so the refresh command has nothing to invalidate and newly written words become visible immediately. - Runtime: ObFTParserProperty now really extracts dict_table / stopword_table / quantifier_table values from the parser-properties JSON (deep-copied into owned buffers), ObFTParseHelper passes them into ObFTIKParam, and ObIKFTParser::init_dict loads a dictionary table (SELECT LOWER(word) ... ORDER BY LOWER(word) COLLATE utf8mb4_bin via inner SQL, then trie -> DAT -> ObFTCacheDict) when the configured table differs from the built-in one, REPLACING the corresponding built-in dictionary. An empty dictionary table maps to the new ObFTEmptyDict which never matches. Covers tools/deploy/mysql_test/test_suite/ai_funcs/t/ik_custom_dict.test: verified byte-exact against the recorded .result (custom words match, words missing from the custom dict degrade to single characters and no longer match, dynamic INSERT + REFRESH takes effect for new rows), and the built-in IK path (no dict_table property) is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Optimize the IK/beng full-text tokenization paths that dominate fulltext index build, TOKENIZE() and MATCH query-token segmentation: - Share the three built-in IK dictionaries (main/quantifier/stopword) process-wide in ObFTParsePluginData: they are immutable, so building the range dict once and borrowing it from every parser instance removes the per-document KV-cache lookups and dictionary object rebuilds that used to run on every tokenization. Custom dictionary tables (FULLTEXT_DICT='Y') keep the per-instance build so REFRESH semantics are unchanged. - Classify each character with a single unicode decode in ObFTCharUtil::do_classify instead of re-decoding for every category check (up to 4 decodes per CJK char before). - Fetch char, length and type in one TokenizeContext::current_char_and_type call in the two per-character hot loops. - Store IK segmenters in a fixed array instead of an allocator-backed list. - Skip the per-token charset conversion (and its allocation) in ObStopWordChecker::check_stopword when source and stopword charsets match. All ai_funcs mysqltest cases still pass byte-exact and tokenization output is unchanged; local fts_large_bench shows build_ik_all -19%, tokenize_ik -37%, tokenize_beng -22% at identical hit counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enable the (previously commented out) check_need_calc_match_score pass in the optimizer: when a MATCH expr is only used as a boolean predicate (e.g. SELECT COUNT(*) ... WHERE MATCH(...) AGAINST(...)), skip the BM25 relevance infrastructure entirely - no total-doc-count aggregation scan, no per-token doc-count aggregation, no forward-index scans and no BM25 evaluation. The match filter then only checks a constant positive hit score. Make the relevance-less plan shape actually executable in the sparse retrieval runtime (these latent null derefs are why the pass had been disabled): - only create/init the doc-cnt agg iter and BM25 estimation context when a doc agg ctdef was generated, and keep the estimator uninitialized when the estimation context is invalid - allocate the skip bit vector unconditionally in the token iter and skip token-doc-cnt estimation without aggregation iters - template TAAT scan ranges from the inverted index scan param when there is no aggregation param, and count dimension hits as 1.0 without a relevance expr - functional lookup / match score output / fts index merge branches always calculate relevance since the score itself is their output All ai_funcs mysqltest cases pass byte-exact, MATCH result sets (natural language, boolean mode, score projection, order-by-score, functional lookup) are unchanged, and fts_large_bench hit counts are identical with query latency down 15-20% locally on top of the tokenization gains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…60367468) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- TokenizeContext::prepare_next_char decodes length, code point and char type in a single utf8 pass (well_formed_len + re-decode before), and caches the code point so letter/quantifier processors check connectors and cn-numbers without decoding the same bytes again - ObIKArbitrator::output_result uses the same single-pass decode for both its scan loops and reuses the decoded code point for the single-cjk ignore check; utf8-encoded surrogates keep their legacy classification - ObAddWord::groupby_word inserts-or-accumulates with one hash lookup via set_or_update instead of get + set - ObFTIndexRowCache keeps one reusable ObFTWordMap across documents so index build no longer creates/destroys a hash map per doc Verified: ai_funcs (load_file/ai_split_document/ik_custom_dict) all pass, fts_large_bench hit counts identical (cn 8001 / beng 11000 / mixed 7332 / limit 20), tokenize_ik 1.00->0.36ms on the local rig.
TOKENIZE() without explicit properties reforms the same default property JSON for a given parser on every statement (json init/parse/rebuild/ serialize). Cache the reformed JSON per parser inner name in a small process-wide table; statements with explicit properties keep the original path. Verified: ai_funcs 3/3 pass, bench hit counts unchanged (cn 8001 / beng 11000 / mixed 7332 / limit 20).
- ObFTParseHelper::init parsed the same property JSON on every call (per statement for TOKENIZE, per helper for index build); cache the parsed ObFTParserProperty keyed by (parser name, property json) in a small process-wide table, falling back to the original parse path when the cache is full or the key is oversized - TOKENIZE per-statement word map sizes buckets at len/4 instead of len/2; load factor stays below one for typical token densities Verified: ai_funcs 3/3 pass, bench hit counts unchanged (cn 8001 / beng 11000 / mixed 7332 / limit 20).
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 100.00 / 100 ik_custom_dict: 100 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 9.07 / 100 mean_improvement: 4.53% full_score_improvement: 50.00% build_improvement: 3.97% build_ik_all_sec: baseline=35.2836, current=32.937, improvement=6.65% build_ik_content_sec: baseline=28.3764, current=26.939, improvement=5.07% build_beng_en_sec: baseline=14.7578, current=14.731, improvement=0.18% tokenize_improvement: -3.24% tokenize_ik_avg_ms: baseline=0.76478, current=0.7473, improvement=2.29% tokenize_beng_avg_ms: baseline=0.42262, current=0.4597, improvement=-8.77% query_improvement: 12.88% query_cn_avg_ms: baseline=16.6628, current=13.9427, improvement=16.32% query_beng_avg_ms: baseline=24.3042, current=20.2865, improvement=16.53% query_mixed_avg_ms: baseline=17.5593, current=14.4142, improvement=17.91% query_limit_avg_ms: baseline=16.2334, current=16.1126, improvement=0.74% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-18 08:00:56 label: vldb-ci-29635600974-1 git_head: 37e0d27 git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2691 select1_stdev_ms: 0.0055 raw_load_sec: 1.483 raw_load_rows_per_sec: 13486.2 build_ik_all_sec: 32.937 build_ik_content_sec: 26.939 build_beng_en_sec: 14.731 build_total_sec: 74.620 ---------------------------------------- tokenize_ik_avg_ms: 0.7473 tokenize_ik_median_ms: 0.7553 tokenize_ik_stdev_ms: 0.0190 tokenize_beng_avg_ms: 0.4597 tokenize_beng_median_ms:0.4441 tokenize_beng_stdev_ms: 0.0232 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 13.9427 query_cn_stdev_ms: 0.0163 query_beng_hits: 11000 query_beng_avg_ms: 20.2865 query_beng_stdev_ms: 0.0214 query_mixed_hits: 7332 query_mixed_avg_ms: 14.4142 query_mixed_stdev_ms: 0.0203 query_limit_hits: 20 query_limit_avg_ms: 16.1126 query_limit_stdev_ms: 0.0058 ======================================== |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task Description
Solution Description
Passed Regressions
Upgrade Compatibility
Other Information
Release Note