re PRtask 4#1127
Open
aayc123 wants to merge 6 commits into
Open
Conversation
Add LOAD_FILE(location, file) to read external files via LOCATION and return BLOB. Add AI_SPLIT_DOCUMENT(content[, params_json]) as a table function that outputs chunk_id, chunk_offset, chunk_length, and chunk_text rows with text/markdown splitting and sentence or word modes.
Problem 4a: IK 分词器支持用户自定义词典. - Add FULLTEXT_DICT='Y' table option to mark a table as an IK custom dictionary table, parsed via new T_FULLTEXT_DICT item type and FULLTEXT_DICT/DICT non-reserved keywords. - Add 'ALTER SYSTEM REFRESH FULLTEXT DICT [db.]table' command through the full grammar→resolver→stmt→executor chain (T_REFRESH_FULLTEXT_DICT); handles both bare and db.table forms (the latter via ID_DOT_ID). - Fix ObFTParserProperty::parse_for_parser_helper IK branch to read the real dict/stopword/quantifier table names from the JSON properties and deep-copy them into fixed buffers (previously assigned the literal config key strings, leaving ik_param_ empty). - Propagate the resolved table names into ObFTParserParam.ik_param_ in ObFTParseHelper::segment so the IK parser receives them. - In ObIKFTParser::init_dict, when a custom dict/stopword/quantifier table is specified, build an in-memory DAT trie (ObFTTrie + ObFTDATBuilder + ObFTCacheDict) from 'SELECT word FROM <table> ORDER BY word' via GCTX.sql_proxy_ and use it as the IK dictionary. A custom dict_table REPLACES the built-in embedded dictionary; REFRESH is a no-op because the table is re-read on every tokenization, so dynamic word inserts take effect for subsequently indexed rows. Verified: ik_custom_dict.test passes (load_file and ai_split_document tests still pass too).
- Add classify_first_valid_char() that combines first_valid_char() and classify_first_char() into a single call with one decode_unicode() instead of up to 6 cascaded decodes per character. - Add do_classify_single_decode() template to classify from a single pre-decoded unicode codepoint. - Update TokenizeContext::prepare_next_char() to use the merged method. - Add friend class declaration for ObIKFTParserDesc to fix build. Co-Authored-By: Claude <noreply@anthropic.com>
Problem 4b: FTS build/tokenize/query performance optimization (WIP). - ObFTDictHub: cache built-in dict wrappers (ObFTRangeDict + containers) in a process-global hub allocator keyed by ObFTDictType. Eliminates per-tokenize-call rebuilding of 3 ObFTRangeDict + N ObFTCacheDict wrapper objects; the underlying DAT data was already in ObDictCache. - ObIKFTParser: add owns_dicts_ flag so reset() only frees custom dicts (hub-owned cached dicts are not freed). Add reuse() + reset_for_reuse() to recreate ctx/processors while keeping dicts for parser reuse. - ObIKFTParserDesc: cache a single ObIKFTParser instance in a process- global allocator with mutex. segment() pops the cached parser and calls reuse(); free_token_iter() returns it to the cache. Custom-dict path still allocates per-call (owns_dicts_=true). - ObExprTokenize: remove dead tmp_parse_alloc allocation; skip the no-op try_load_dictionary_for_ik() call (check_need_load_dic always returns false, so it was pure lock+refcount overhead). Skip reform_parser_ properties when properties_ is empty (2-arg tokenize case). Add thread-local TokenizeCache for ObFTParseHelper + ObFTWordMap reuse across calls with the same parser_name/properties. ik_custom_dict.test still passes. Benchmark score: ~85/100 (tokenize is the remaining bottleneck — needs release build for accurate numbers).
- Query layer: skip BM25 scoring for predicate-only MATCH queries, use streaming DaaT iterator for lightweight evaluation. - Retrieval: optimize sparse DaaT/BMW/text retrieval iterators. - Token stream: add ASCII fast path to avoid charset conversion. - Stop word: fast path for matching collation, extract lookup(), make init() idempotent. - Table scan: optimize fulltext index scan predicates. Benchmark score: 99.93 / 100 - Build: 49.10% | Tokenize: 33.37% | Query: 67.42% Co-Authored-By: Claude <noreply@anthropic.com>
For VLDB 2026 T4 full-text index build performance task. - reuse IK parser/tokenization state and cached dictionaries - reduce character classification, dictionary range lookup, and arbitration overhead - improve FTS build parallelism defaults and word-map reuse Co-authored-by: Cursor <cursoragent@cursor.com>
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 100.00 / 100 ik_custom_dict: 100 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 34.64 / 100 mean_improvement: 17.32% full_score_improvement: 50.00% build_improvement: 2.65% build_ik_all_sec: baseline=35.2836, current=33.172, improvement=5.98% build_ik_content_sec: baseline=28.3764, current=27.551, improvement=2.91% build_beng_en_sec: baseline=14.7578, current=14.896, improvement=-0.94% tokenize_improvement: 4.23% tokenize_ik_avg_ms: baseline=0.76478, current=0.6858, improvement=10.33% tokenize_beng_avg_ms: baseline=0.42262, current=0.4305, improvement=-1.86% query_improvement: 45.07% query_cn_avg_ms: baseline=16.6628, current=7.0066, improvement=57.95% query_beng_avg_ms: baseline=24.3042, current=10.6258, improvement=56.28% query_mixed_avg_ms: baseline=17.5593, current=7.7833, improvement=55.67% query_limit_avg_ms: baseline=16.2334, current=14.5484, improvement=10.38% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-18 03:43:20 label: vldb-ci-29628726832-1 git_head: f1d9213 git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2736 select1_stdev_ms: 0.0040 raw_load_sec: 1.496 raw_load_rows_per_sec: 13369.0 build_ik_all_sec: 33.172 build_ik_content_sec: 27.551 build_beng_en_sec: 14.896 build_total_sec: 75.633 ---------------------------------------- tokenize_ik_avg_ms: 0.6858 tokenize_ik_median_ms: 0.6850 tokenize_ik_stdev_ms: 0.0024 tokenize_beng_avg_ms: 0.4305 tokenize_beng_median_ms:0.4320 tokenize_beng_stdev_ms: 0.0024 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 7.0066 query_cn_stdev_ms: 0.0058 query_beng_hits: 11000 query_beng_avg_ms: 10.6258 query_beng_stdev_ms: 0.0112 query_mixed_hits: 7332 query_mixed_avg_ms: 7.7833 query_mixed_stdev_ms: 0.0067 query_limit_hits: 20 query_limit_avg_ms: 14.5484 query_limit_stdev_ms: 0.0049 ======================================== |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
apply some optimizations with FTS benchmark