feat: support custom dictionaries for IK parser#1144
Open
Ludwig-LMY wants to merge 14 commits into
Open
Conversation
Adds the Document AI (load_file, ai_split_document, ai_parse_doc) and custom IK dictionary mysqltest cases under tools/deploy/mysql_test/test_suite/ai_funcs, plus the vldb.yml CI workflow that builds the observer and runs the suite. The function implementations are tracked separately on the vldb_impl branch.
…ceanbase#1009) ai_parse_doc needs a live vision endpoint, so remove it from the CI test loop and delete the hardcoded API_KEY/MODEL_NAME/API_ENDPOINT env; the remaining three cases (load_file, ai_split_document, ik_custom_dict) run credential-free.
* CI: score the ai_funcs results After the ai_funcs mysqltests run, emit two score reports in the same format as the FTS large benchmark (title, = underline, then score: X.XX / 100): Document AI Functions = load_file (50) + ai_split_document (50) IK Custom Dictionary = ik_custom_dict (100) A passing case earns its full points, a failing one earns zero. The reports are written to ai_funcs_score.txt and appended to the CI step summary. * CI: drop the ai_funcs score step-summary block The score report already prints to the step log and ai_funcs_score.txt, so the GitHub step-summary duplication is unnecessary. * CI: drop the observer log tail from the ai_funcs step The per-case PASS/FAIL lines and the score report are enough; the seekdb.log tail on failure is noise here. * CI: print the ai_funcs score report directly to the step log Nothing reads ai_funcs_score.txt now that the step-summary block is gone, so drop SCORE_OUTPUT and the tee and just echo the report.
* CI: do not fail the ai_funcs step on a failing case Drop the explicit exit so the step always reports the score. A failing case now lowers its score to zero instead of failing the workflow. * CI: move the ai_funcs score report to the comment-artifact step The mysqltest step now writes ai_funcs_score.txt instead of printing the report. Prepare FTS benchmark comment artifact prints it and ships it in the uploaded artifact alongside the FTS score. * CI: post the ai_funcs scores in the PR benchmark comment Prepare FTS benchmark comment artifact now stages ai_funcs_score.txt into the uploaded artifact, and only skips when neither the benchmark report nor the ai_funcs score exists. vldb-benchmark-comment renders that file as a "Document AI & IK Custom Dictionary Score" section above the FTS sections. Each section is emitted only when its input file is present, so a run without the FTS benchmark still posts the ai_funcs scores. * CI: drop the misleading FTS prefix from the comment-artifact steps Those steps and their staging directory now carry the ai_funcs scores as well as the FTS benchmark, so name them for what they are. The Prepare step locals become FTS_REPORT/FTS_SCORE so they read clearly beside AI_FUNCS_SCORE, and the staging directory becomes vldb_comment. The artifact name and the file names inside it are a contract with vldb-benchmark-comment.yml, which runs from the default branch. Renaming one side alone would silently drop the PR comment, so they are left unchanged and now carry comments saying so.
load_file no longer runs the load_file -> AI_SPLIT_DOCUMENT integration query (ai_split_document has its own case), so its ai_lf_doc.txt fixture goes too. ik_custom_dict drops the two TOKENIZE previews and the stopword/quantifier tables they created; the case keeps the dictionary-driven MATCH coverage. Both .result files are updated to match, and all three ai_funcs cases pass.
(cherry picked from commit d622c5e)
|
wangzelin.wzl seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Support custom full-text dictionaries for the IK parser in SeekDB.
This includes dictionary table support, refresh handling, parser properties, TOKENIZE integration, full-text index behavior, and dynamic dictionary update validation.
Validation
/root/seekdb-ik-custom-dict-buildbuild_debug/src/observer/seekdb2881tools/deploy/mysql_test/test_suite/ai_funcs/t/ik_custom_dict.testokNotes
Also includes a small build compatibility fix for locating
libaio.so.1t64correctly on the WSL environment.