fix: preserve HME100k prediction case in OCRBench scoring#1278
fix: preserve HME100k prediction case in OCRBench scoring#1278kcz358 merged 1 commit intoEvolvingLMMs-Lab:mainfrom
Conversation
kcz358
left a comment
There was a problem hiding this comment.
Hi, so instead of just remove the lower(), maybe should actually lower the gt_ans as well?
@kcz358 I don't think we should lowercase HME100k should be case-sensitive, since it is handwritten mathematical expression recognition. So, we should not lowercase Since these answers are math-expression / LaTeX-like strings, lowercasing |
|
Got it, looks make sense if HME requires the answer to be case sensitive. Will this change cause false negative on other branches? If not then I will merge this PR. Thanks |
|
No, this should not affect the other OCRBench branches. |
Summary
Why
Issue #1220 points out that
ocrbench_process_resultslowercasespredbefore branching ondataset_name, while theHME100kbranch intentionally compares answers without lowercasing them. That makes correct HME100k predictions score as 0 when the only difference is letter case.Testing
Closes #1220