perf(fts): integrate reference-aligned hot paths#1141
Open
StarAstronaut2 wants to merge 22 commits into
Open
Conversation
1. ObStopWordChecker: reuse scratch ArenaAllocator via ObAddWord member, avoid per-word allocator create/destroy overhead. 2. ObStopWordChecker: skip charset_convert when word and stopword collation types match (fast path for UTF8MB4_GENERAL_CI). 3. ObIKArbitrator: add reuse() and make it a member of ObIKFTParser, avoid per-batch hashmap create/destroy. 4. TokenizeContext: merge current_char() + current_char_type() into current_char_and_type(), halving per-character function calls on the tokenization hot path. Benchmark (release build, fts_large_bench.sh vs CI baseline): - build: +65% (build_ik_all: 12.2s vs 35.3s) - tokenize: +64% (tokenize_ik: 0.288ms vs 0.765ms) - query: +57% (query_cn: 6.67ms vs 16.66ms) - Score: 100 / 100 Co-Authored-By: 糊涂涂 <hututu@seekdb.dev>
This reverts commit 24538a9.
|
lzr seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 0.00 / 100 ik_custom_dict: 0 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 100.00 / 100 mean_improvement: 54.39% full_score_improvement: 50.00% build_improvement: 39.63% build_ik_all_sec: baseline=35.2836, current=19.181, improvement=45.64% build_ik_content_sec: baseline=28.3764, current=18.038, improvement=36.43% build_beng_en_sec: baseline=14.7578, current=9.323, improvement=36.83% tokenize_improvement: 40.40% tokenize_ik_avg_ms: baseline=0.76478, current=0.3051, improvement=60.11% tokenize_beng_avg_ms: baseline=0.42262, current=0.3352, improvement=20.69% query_improvement: 83.13% query_cn_avg_ms: baseline=16.6628, current=1.6068, improvement=90.36% query_beng_avg_ms: baseline=24.3042, current=9.4101, improvement=61.28% query_mixed_avg_ms: baseline=17.5593, current=1.6224, improvement=90.76% query_limit_avg_ms: baseline=16.2334, current=1.6018, improvement=90.13% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-18 08:05:52 label: vldb-ci-29636136041-1 git_head: ab23ef8 git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2718 select1_stdev_ms: 0.0108 raw_load_sec: 1.508 raw_load_rows_per_sec: 13262.6 build_ik_all_sec: 19.181 build_ik_content_sec: 18.038 build_beng_en_sec: 9.323 build_total_sec: 46.554 ---------------------------------------- tokenize_ik_avg_ms: 0.3051 tokenize_ik_median_ms: 0.3025 tokenize_ik_stdev_ms: 0.0056 tokenize_beng_avg_ms: 0.3352 tokenize_beng_median_ms:0.3289 tokenize_beng_stdev_ms: 0.0093 ---------------------------------------- query_cn_hits: 0 query_cn_avg_ms: 1.6068 query_cn_stdev_ms: 0.0019 query_beng_hits: 11000 query_beng_avg_ms: 9.4101 query_beng_stdev_ms: 0.0135 query_mixed_hits: 0 query_mixed_avg_ms: 1.6224 query_mixed_stdev_ms: 0.0114 query_limit_hits: 0 query_limit_avg_ms: 1.6018 query_limit_stdev_ms: 0.0029 ======================================== |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
This PR is intentionally separated from #1081 so that #1081 remains pinned to its authoritative 19.33-point commit (
074fb7c2).