Use Intl.Segmenter for the entire-word search matching

[calixteman]

getCharacterType was a port of the old WordBreaker::GetClass, which has been removed and replaced with ICU4X word segmentation. Use Intl.Segmenter instead, testing each match boundary on the two adjacent grapheme clusters in isolation like Firefox's find.

It fixes #21520.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.41%. Comparing base (330cc4f) to head (0c97aa7).

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #21521   +/-   ##
=======================================
  Coverage   89.41%   89.41%           
=======================================
  Files         262      262           
  Lines       66738    66704   -34     
=======================================
- Hits        59675    59646   -29     
+ Misses       7063     7058    -5     

Flag	Coverage Δ
browsertest	66.56% <18.18%> (+0.05%)	⬆️
integrationtest	68.77% <18.18%> (+0.04%)	⬆️
unittest	57.31% <100.00%> (-0.03%)	⬇️
unittestcli	56.26% <100.00%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[markmatney]

Thank you for this @calixteman -- I've tested this locally (in both Firefox and Chromium) and the adjacent dash case appears to be fixed. Here's what I'm seeing for the other two cases:

• superscript: a whole-word search still isn't matching the occurrence of property adjacent to the superscript on pg 32; but a search for property2 is matching that way, and it wasn't before (it used to only if Whole Words was disabled); still not what I'd expect, but perhaps a useful data point to be aware of. Is this working as expected on your machine, though?
• hyphenation across line break: search yields 1 match for intellectual-property, 1 match for intellectualproperty; likewise for special-interest and specialinterest (and for each query, the match count is the same with or without Whole Words enabled). Matching on these incorrect un-hyphenated forms was actually the existing behavior, which I didn't notice before. Chromium's implementation doesn't have whole word search, but otherwise exhibits the same odd behavior. With mupdf a search for the (correct) hyphenated examples yields 2 matches as expected.

[calixteman]

Yep I saw these two issues.
The superscript one is the more complex one. In the content stream it's just a small raised "2", so from a pure text point of view it's just a "2". But even if we'd have a "²", it'd have failed because it's normalized as a "2" (people will likely search "2" and not "²"). I think it's more or less the same issue in Firefox: the "²" is, in the Wikipedia page, a <sup>2</sup> some from a basic point of view, the text is "this2" and not "this²".
About the hyphenation: I don't really know what to think about it. I speak English (more or less) and I know (I guess in fact because I'm not a native speaker) that it isn't an hyphenation. If we use "-" as a delimiter then "inter" will be found in "inter-\nnational" in whole-word mode. So I think this is a context dependent problem. As far as I know there is API in JS for guessing such a thing.
So for now I don't have any good solutions.

[markmatney]

IMO, for the whole-words search function it would be better to allow some false positives than false negatives (if those are the two choices) because I think it's always easier (and better) for the user to review and discard false positives than it is for them to find the false negatives; I spent the better part of an hour binary-searching the example PDF while toggling the Whole Words option in order to locate the missing occurrences. Of course, I only did that because I knew they existed; most users would be none the wiser and would simply not find what they were looking for (at unknown cost), unless they disabled Whole Words and dug through the larger set of matches. However, that larger set may be much larger indeed. The example query "property" is not particularly illustrative here, since almost all words containing it as a substring have something to do with "property" the concept, but consider a word like "art"; there are many words containing that substring that have nothing at all to do with "art" the concept, evidenced by the 421 occurrences of that substring in the Free Culture text when Whole Words is disabled, but only 12 occurrences otherwise.

For that reason, I think it makes sense to consider property2 to contain a word boundary between the y and the 2, and likewise for hyphenated words with a line break. Although, my perspective is that of a monolingual English speaker, and I am not aware of the specific contexts where this would be inappropriate.

Also I may not be clear on what you mean by "it isn't an hyphenation", but I am pretty sure that "intellectual-property" and "special-interest" are correctly hyphenated in the Free Culture text, since they are being used as adjectives rather than nouns.

[timvandermeij]

Looks good, with the comment addressed. Good idea to use the built-in Intl.Segmenter for this now that it is broadly available because it makes the logic quite a bit simpler!