合并字符+内存分离#1128
Open
yolo148 wants to merge 14 commits into
Open
Conversation
Implement the two Document AI functions for the ai_funcs test suite: LOAD_FILE(location_name, file_name) -> BLOB - new scalar sys expr ObExprLoadFile (T_FUN_SYS_LOAD_FILE): looks up the LOCATION schema by name, joins its URL with the file name and reads the file through ObBackupIoAdapter, returning the content as a binary lob AI_SPLIT_DOCUMENT(content [, parameters]) table function - new non-reserved keyword + grammar rule producing T_AI_SPLIT_DOCUMENT_EXPRESSION, resolved as a JSON_TABLE-family table item (OB_AI_SPLIT_DOC_TABLE_TYPE) with four fixed output columns chunk_id / chunk_offset / chunk_length / chunk_text - AiSplitDocTableFunc splits the document per parameters json: type text|markdown (default markdown), by word|sentence (default word), max units per chunk (default 256), overlap for sliding windows; markdown mode sections the text at '#' heading lines and prefixes each chunk with its section heading - spec serialization, printer (view expansion) and rescan support Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implement the Custom Dictionary feature for the IK fulltext parser: - CREATE TABLE ... FULLTEXT_DICT='Y' table option: new FULLTEXT_DICT non-reserved keyword and table option, accepted for CREATE TABLE (value must be 'Y'/'N', rejected in ALTER TABLE like ORGANIZATION). - ALTER SYSTEM REFRESH FULLTEXT DICT [db.]table statement: new DICT keyword, T_REFRESH_FULLTEXT_DICT parse node, resolver, stmt and executor wired through the standard system-command chain. Accepts the unquoted db.table / table forms and the double-quoted string forms. Custom dictionaries are re-read from the dictionary table on every parser instantiation, so the refresh command has nothing to invalidate and newly written words become visible immediately. - Runtime: ObFTParserProperty now really extracts dict_table / stopword_table / quantifier_table values from the parser-properties JSON (deep-copied into owned buffers), ObFTParseHelper passes them into ObFTIKParam, and ObIKFTParser::init_dict loads a dictionary table (SELECT LOWER(word) ... ORDER BY LOWER(word) COLLATE utf8mb4_bin via inner SQL, then trie -> DAT -> ObFTCacheDict) when the configured table differs from the built-in one, REPLACING the corresponding built-in dictionary. An empty dictionary table maps to the new ObFTEmptyDict which never matches. Covers tools/deploy/mysql_test/test_suite/ai_funcs/t/ik_custom_dict.test: verified byte-exact against the recorded .result (custom words match, words missing from the custom dict degrade to single characters and no longer match, dynamic INSERT + REFRESH takes effect for new rows), and the built-in IK path (no dict_table property) is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Optimize the IK/beng full-text tokenization paths that dominate fulltext index build, TOKENIZE() and MATCH query-token segmentation: - Share the three built-in IK dictionaries (main/quantifier/stopword) process-wide in ObFTParsePluginData: they are immutable, so building the range dict once and borrowing it from every parser instance removes the per-document KV-cache lookups and dictionary object rebuilds that used to run on every tokenization. Custom dictionary tables (FULLTEXT_DICT='Y') keep the per-instance build so REFRESH semantics are unchanged. - Classify each character with a single unicode decode in ObFTCharUtil::do_classify instead of re-decoding for every category check (up to 4 decodes per CJK char before). - Fetch char, length and type in one TokenizeContext::current_char_and_type call in the two per-character hot loops. - Store IK segmenters in a fixed array instead of an allocator-backed list. - Skip the per-token charset conversion (and its allocation) in ObStopWordChecker::check_stopword when source and stopword charsets match. All ai_funcs mysqltest cases still pass byte-exact and tokenization output is unchanged; local fts_large_bench shows build_ik_all -19%, tokenize_ik -37%, tokenize_beng -22% at identical hit counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enable the (previously commented out) check_need_calc_match_score pass in the optimizer: when a MATCH expr is only used as a boolean predicate (e.g. SELECT COUNT(*) ... WHERE MATCH(...) AGAINST(...)), skip the BM25 relevance infrastructure entirely - no total-doc-count aggregation scan, no per-token doc-count aggregation, no forward-index scans and no BM25 evaluation. The match filter then only checks a constant positive hit score. Make the relevance-less plan shape actually executable in the sparse retrieval runtime (these latent null derefs are why the pass had been disabled): - only create/init the doc-cnt agg iter and BM25 estimation context when a doc agg ctdef was generated, and keep the estimator uninitialized when the estimation context is invalid - allocate the skip bit vector unconditionally in the token iter and skip token-doc-cnt estimation without aggregation iters - template TAAT scan ranges from the inverted index scan param when there is no aggregation param, and count dimension hits as 1.0 without a relevance expr - functional lookup / match score output / fts index merge branches always calculate relevance since the score itself is their output All ai_funcs mysqltest cases pass byte-exact, MATCH result sets (natural language, boolean mode, score projection, order-by-score, functional lookup) are unchanged, and fts_large_bench hit counts are identical with query latency down 15-20% locally on top of the tokenization gains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…60367468) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- TokenizeContext::prepare_next_char decodes length, code point and char type in a single utf8 pass (well_formed_len + re-decode before), and caches the code point so letter/quantifier processors check connectors and cn-numbers without decoding the same bytes again - ObIKArbitrator::output_result uses the same single-pass decode for both its scan loops and reuses the decoded code point for the single-cjk ignore check; utf8-encoded surrogates keep their legacy classification - ObAddWord::groupby_word inserts-or-accumulates with one hash lookup via set_or_update instead of get + set - ObFTIndexRowCache keeps one reusable ObFTWordMap across documents so index build no longer creates/destroys a hash map per doc Verified: ai_funcs (load_file/ai_split_document/ik_custom_dict) all pass, fts_large_bench hit counts identical (cn 8001 / beng 11000 / mixed 7332 / limit 20), tokenize_ik 1.00->0.36ms on the local rig.
TOKENIZE() without explicit properties reforms the same default property JSON for a given parser on every statement (json init/parse/rebuild/ serialize). Cache the reformed JSON per parser inner name in a small process-wide table; statements with explicit properties keep the original path. Verified: ai_funcs 3/3 pass, bench hit counts unchanged (cn 8001 / beng 11000 / mixed 7332 / limit 20).
- ObFTParseHelper::init parsed the same property JSON on every call (per statement for TOKENIZE, per helper for index build); cache the parsed ObFTParserProperty keyed by (parser name, property json) in a small process-wide table, falling back to the original parse path when the cache is full or the key is oversized - TOKENIZE per-statement word map sizes buckets at len/4 instead of len/2; load factor stays below one for typical token densities Verified: ai_funcs 3/3 pass, bench hit counts unchanged (cn 8001 / beng 11000 / mixed 7332 / limit 20).
The TAAT doc-dedup hash maps were created with 10 buckets while each partition is designed to hold about 1000 docs (and a single partition holds every hit when relevance calculation is skipped), which forced hundred-deep collision chains on every per-hit upsert. Size buckets to 2x the designed partition size, upsert with one set_or_update pass instead of get + set, and use NoPthreadDefendMode since the maps are only touched by the executing query thread. Verified: ai_funcs 3/3 pass, bench hit counts unchanged (cn 8001 / beng 11000 / mixed 7332 / limit 20).
|
|
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 100.00 / 100 ik_custom_dict: 100 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 10.61 / 100 mean_improvement: 5.30% full_score_improvement: 50.00% build_improvement: 3.51% build_ik_all_sec: baseline=35.2836, current=32.541, improvement=7.77% build_ik_content_sec: baseline=28.3764, current=26.81, improvement=5.52% build_beng_en_sec: baseline=14.7578, current=15.164, improvement=-2.75% tokenize_improvement: -0.90% tokenize_ik_avg_ms: baseline=0.76478, current=0.7191, improvement=5.97% tokenize_beng_avg_ms: baseline=0.42262, current=0.4555, improvement=-7.78% query_improvement: 13.30% query_cn_avg_ms: baseline=16.6628, current=13.8449, improvement=16.91% query_beng_avg_ms: baseline=24.3042, current=20.1079, improvement=17.27% query_mixed_avg_ms: baseline=17.5593, current=14.3043, improvement=18.54% query_limit_avg_ms: baseline=16.2334, current=16.1529, improvement=0.50% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-18 06:38:41 label: vldb-ci-29633569432-1 git_head: c0527b0 git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2670 select1_stdev_ms: 0.0022 raw_load_sec: 1.505 raw_load_rows_per_sec: 13289.0 build_ik_all_sec: 32.541 build_ik_content_sec: 26.810 build_beng_en_sec: 15.164 build_total_sec: 74.528 ---------------------------------------- tokenize_ik_avg_ms: 0.7191 tokenize_ik_median_ms: 0.7181 tokenize_ik_stdev_ms: 0.0017 tokenize_beng_avg_ms: 0.4555 tokenize_beng_median_ms:0.4487 tokenize_beng_stdev_ms: 0.0108 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 13.8449 query_cn_stdev_ms: 0.0244 query_beng_hits: 11000 query_beng_avg_ms: 20.1079 query_beng_stdev_ms: 0.0339 query_mixed_hits: 7332 query_mixed_avg_ms: 14.3043 query_mixed_stdev_ms: 0.0012 query_limit_hits: 20 query_limit_avg_ms: 16.1529 query_limit_stdev_ms: 0.0343 ======================================== |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task Description
Solution Description
Passed Regressions
Upgrade Compatibility
Other Information
Release Note