grhoten · 2026-06-27T00:59:41Z

Fixes #188
Fixes #187
Fixes #186
Fixes #185
Fixes #183
Fixes #178
Fixes #117
Fixes #113
Fixes #87

The changes address the following issues.

The existing lexical dictionaries are updated.
Language support
- 23 new languages are being supported.
- 3 languages perform longest suffix guessing when a word is missing from a lexical dictionary.
- Several tests have been commented out due to missing lexeme data from Wikidata.
- Several languages have adopted the new RBNF changes for improved grammatical agreement of quantities. The HEAD changes from CLDR are required for these changes, which requires the upcoming ICU 79 release.
- A lot of data is being added.
Various performance improvements. Some changes are aligned with ICU performance improvements.
The structures are data aligned. It follows the ICU design principle for data alignment. This is a binary breaking change, which is OK for a pre-1.0 release.
There is a blind port to Windows. It’s untested, but it minimizes the effort to support testing on Windows in GitHub.
Message Format 2 support significantly updated.
- The old data driven FormatTest has been converted into Message2Test
- Support added for InflectableStringConcept, SemanticConcept, SemanticConceptList, PronounConcept, NumberConcept, and quantify have been added.
- Support for SSML for conversion from SpeakableString. This is the best way to shoehorn in the functionality.
- This coverage is mostly complete. This support should probably be reviewed with the MF2 team.
- This is an API breaking change, which is OK for a pre-1.0 release.
The dictionary-parser now has passing tests.
The Serbian and Malayalam grammar synthesizers are roughly the same, but they could adopt some new internal frameworks.
And of course various bug fixes.

The version of ICU needs to be updated before the pull request checks can pass. The Language List should be updated once this pull request is merged.

grhoten

Here are some highlights of these changes.

grhoten · 2026-06-27T01:11:31Z

+
+// Croatian (hr) and Serbian (sr) share Bosnian-Croatian-Serbian numeral-noun agreement: the 2-4
+// paucal takes the genitive singular and 5+ takes the genitive plural.
+SerboCroatianCommonConceptFactory::SerboCroatianCommonConceptFactory(const ::inflection::util::ULocale& language)


@nciric You may be interested in this.

Is there something I need to change or is Serbian going to pick it up by default (I can look at Croatian for guidance).

I don't think that there is anything left to change in this class. Croatian is using the same class now. Both are registered in LocalizedCommonConceptFactoryProvider.cpp.

grhoten · 2026-06-27T01:13:43Z

@@ -0,0 +1,195 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>


The data driven tests in this directory has a lot of examples of how to integrate Unicode Inflection into MF2.

grhoten · 2026-06-27T01:15:48Z

+        <params>
+            <param name="object" type="String"><value>lights</value></param>
+        </params>
+        <source>The {$object} {is :inflect number=$object} on.</source>
+        <print>The lights are on.</print>


This is how you inflect a word based on another word.

grhoten · 2026-06-27T01:16:54Z

+        <params>
+            <param name="object" type="String"><value>lights</value></param>
+        </params>
+        <source>{$object :inflect definiteness=definite} {is :inflect number=$object} on.</source>
+        <print>the lights are on.</print>
+    </test>


Add "the" in front of a word. This is boring in English. It's more interesting in the rest of the European languages.

grhoten · 2026-06-27T01:17:46Z

+        <params>
+            <param name="object" type="String"><value>light</value></param>
+        </params>
+        <source>.local $num = {$object :inflect to=number} .match $num
+plural {{The {$object} are on}}
+* {{The {$object} is on}}
+</source>
+        <print>The light is on</print>
+    </test>


This is the verbose way to implement this functionality. I'm not a fan of it, but it shows that the property can be queried for a more complex phrase.

grhoten · 2026-06-27T05:07:46Z

+    const inflection::dialog::SemanticFeatureModel* currModel = nullptr;
+    std::atomic finished(false);
+    std::barrier barrier(processorCount + 1);
+    for (int32_t count = 0; count < processorCount; count++) {
+        results[count].reserve(minimumNumberOfWords);
+        threads.emplace_back(testInflection, &currModel, &phrases, &constraints, &semanticFeatures, &barrier, &finished, count, &results[count]);
+    }


We're reusing the threads and modernizing the style of thread safety checking.

grhoten · 2026-06-27T05:11:19Z

+ * @brief Provides utility functions for use arrays.
+ * @details Provides utility functions for use arrays.
+ */
+class inflection::util::ArrayUtils {


This helps reduce the heap usage a fair amount. This allows you to perform a binary search in an arbitrary sorted constexpr C data array, and it's verified as sorted at compile time. You get a compiler error if it's unsorted.

grhoten · 2026-06-27T05:15:56Z

@@ -4,104 +4,112 @@
 #pragma once

 #include <inflection/grammar/synthesis/fwd.hpp>
-#include <string>

 class inflection::grammar::synthesis::GrammemeConstants final


This is now a header only class. It avoids permanently caching these strings on the heap, and there is no thread safe initialization to be concerned about.

grhoten · 2026-06-27T05:19:20Z

+       * icu::messsage2::MFFunctionRegistry::Builder::adoptFormatter
+       * to register a custom formatter factory.
+       */
+      static icu::message2::FormatterFactory* createInflectFormatterFactory(OutputMode mode = OutputMode::PLAIN);


By default, we format as plain text. If you want to retain the spoken form, then it can be outputted as SSML. This is an API breaking change.

As a reminder, this is a pre-1.0 release. So API stability guarantees are not there yet.

grhoten · 2026-06-27T05:22:25Z

-        OUTPUT_FILE /dev/stdout
-        ERROR_FILE /dev/stderr
+
+if(WIN32)


If someone could run this on Windows, that would be helpful. The default is to print in color, but on Windows, we don't run the sh shell. I don't know if this test runs in color. This can be verified by manually running this from the command line outside of the makefiles.

grhoten · 2026-06-27T05:42:49Z

+    if (toFormat.getType() == UFMT_DOUBLE) {
+        num = std::make_unique<inflection::dialog::NumberConcept>(
+            toFormat.getDouble(errorCode), locale, locale);
+    }
+    else {
+        num = std::make_unique<inflection::dialog::NumberConcept>(
+            toFormat.getInt64(errorCode), locale, locale);
+    }


We recently had a meeting in ICU around this topic. There are times that formatting a number and speaking a number need to be a bit different in such scenarios. This should not be doubling up the locale usage like this. When formatting context allows this value to be decoupled, we should modify these lines.

This is a temporary workaround for now.

grhoten · 2026-06-27T14:04:46Z

+    <!-- FIXME: <test><source number="singular" case="illative">Lahti</source><result>Lahteen</result></test> -->
+    <!-- FIXME: <test><source number="singular" case="adessive">Lahti</source><result>Lahdella</result></test> -->
+    <!-- FIXME: <test><source number="singular" case="ablative">Lahti</source><result>Lahdelta</result></test> -->
+    <!-- FIXME: <test><source number="singular" case="allative">Lahti</source><result>Lahdelle</result></test> -->


Due to the lack of Wikidata lexeme coverage, there are numerous tests that are failing, especially for the new languages. These tests have a FIXME tag on each failing test. There are over a thousand of these data coverage issues, and the vast majority of them are in Finnish.

For all languages, adding/fixing the missing data is usually the best way to fix the issue.

For Finnish, Polish, and Russian, adding the missing words mentioned in the suffix tables will allow numerous words to be supported en masse. Addressing that data coverage issue would allow the Exemplar test to be enabled for that language.

Occasionally, the issue is not a data coverage issue, and it's an algorithm or data structure issue. There are a handful of these issues. The Norwegian and Dutch failures may fall into this category, but I'm not sure yet.

nciric

qq about "The existing lexical dictionaries are updated."

I assume no lexemes were added to the dictionaries that don't already exist in Wikidata. If that's not the case, we'll have a problem next time we sync with Wikidata as it will override items.

grhoten · 2026-07-08T18:07:39Z

qq about "The existing lexical dictionaries are updated."

I assume no lexemes were added to the dictionaries that don't already exist in Wikidata. If that's not the case, we'll have a problem next time we sync with Wikidata as it will override items.

I don't quite understand the question. I did not manually edit the data derived from Wikidata, if that is what you mean.

The data is derived like this, and this is how you update the lexical dictionaries.

cd inflection/tools/dictionary-parser
./ParseWikidata --all ../../../wikidata-20260708-lexemes.json

Note: I have to update this data once more before the final review.

nciric · 2026-07-08T18:05:06Z

-    return {inflectedWord};
+    static constexpr const char16_t* const doNotInflect[] = {
+        // This must be in sorted order.
+        u"августа",


Why are month names hardcoded?

That's a good question. I think this may be a misnomer from a previous version of this code.

It should probably be renamed as isNotGuessable or something like that because it's only used in guessInflection. Though if the words are complete, then it's just dead code/data. This needs to be revisited before merging.

The overall change looks good - once this is fixed (and maybe Romanian dictionary) feel free to land.

After further review, I think this is fine as is. It looks like if you try to change these genitive forms to accusative for dates, they stay unchanged according to one of the tests. Since this part is refactor, and it was already here, I recommend leaving this as-is. I'm fine with revisiting this separately with a Russian expert.

nciric · 2026-07-08T18:14:43Z


 bool Inflector_InflectionPattern::containsSuffix(std::u16string_view suffix) const
 {
+    const auto suffixID = inflectorDictionary.inflectionSuffixes.getIdentifierIfAvailable(suffix);


This is where we "guess" word inflection if not in dictionary by suffix in inflectional_xx.xml?

No, check Polish, Finnish, and Russian. They use a suffix file of exemplar words. It's mostly automatically generated. Some manual edits are performed on the derived suffix files. Unfortunately, the suffix files are from a more complete lexical dictionary. They should probably be rederived from Wikidata, but that should wait till after the data in Wikidata has better coverage. The ExemplarTest#testExemplarsInDictionary test has a TODO to work on this task.

Right now, it has partial word coverage, which is why the ExemplarTest#testExemplarsInDictionary test fails.

nciric · 2026-07-08T18:21:00Z

+
+// Croatian (hr) and Serbian (sr) share Bosnian-Croatian-Serbian numeral-noun agreement: the 2-4
+// paucal takes the genitive singular and 5+ takes the genitive plural.
+SerboCroatianCommonConceptFactory::SerboCroatianCommonConceptFactory(const ::inflection::util::ULocale& language)


Is there something I need to change or is Serbian going to pick it up by default (I can look at Croatian for guidance).

nciric · 2026-07-08T18:24:05Z

qq about "The existing lexical dictionaries are updated."
I assume no lexemes were added to the dictionaries that don't already exist in Wikidata. If that's not the case, we'll have a problem next time we sync with Wikidata as it will override items.

I don't quite understand the question. I did not manually edit the data derived from Wikidata, if that is what you mean.

The data is derived like this, and this is how you update the lexical dictionaries.
cd inflection/tools/dictionary-parser
./ParseWikidata --all ../../../wikidata-20260708-lexemes.json
Note: I have to update this data once more before the final review.

I understand the process, my questions was about data source:

Is it all from wikidata
Or some of it was directly added to the dictionary without being added to wikidata

grhoten · 2026-07-08T18:38:01Z

my questions was about data source:

Is it all from wikidata

Or some of it was directly added to the dictionary without being added to wikidata

It's from Wikidata with some filtering and augmentation from ParseWikidata (dictionary-parser). That tool does augment some data that isn't in Wikidata, like adding sound properties or some other grammemes. The filter properties there does reorder, remove, and merge some lexemes.

It's all automated. No manual edits were performed.

The sound properties could be included into Wikidata by adding phonetic information to many words. That's hard to derive when you only need a subset of the data. So right now, that data is primarily separate. If Wikidata has phonetic information on a form, it will be parsed and the tool will derive the important information.

grhoten · 2026-07-08T18:52:16Z

Is there something I need to change or is Serbian going to pick it up by default (I can look at Croatian for guidance).

I recommend reviewing QuantifyTest#testSerboCroatian for accuracy. If there are some grammatical cases besides the nominative form that you want to add, those grammatical case tests can be added. You can look at Russian for some examples.

The SerboCroatianCommonConceptFactory class is registered in LocalizedCommonConceptFactoryProvider for Serbian. It's not automatically registered.

Inflection-187 Support SemanticConcept in icu::message2 for custom vocabulary Inflection-186 Support the print and speak lines from SpeakableString in icu::message2 Inflection-185 Support lists in icu::message2 Inflection-183 Support quantify in icu::message2 Inflection-178 Provide quantity support for Serbian Inflection-117 Support inflection functionality into icu::message2 Inflection-113 Derive sound properties from Wikidata for Korean and Turkish Inflection-87 Integrate Unicode Inflection into Unicode Message Format Support more languages

grhoten commented Jun 27, 2026

View reviewed changes

grhoten force-pushed the main branch from 71dd5d9 to 5894887 Compare June 28, 2026 14:25

nciric reviewed Jul 8, 2026

View reviewed changes

nciric approved these changes Jul 8, 2026

View reviewed changes

nciric approved these changes Jul 10, 2026

View reviewed changes

grhoten force-pushed the main branch 2 times, most recently from 01e0926 to 41b1ac4 Compare July 10, 2026 22:22

grhoten force-pushed the main branch from bd81502 to 384b498 Compare July 12, 2026 22:22

grhoten merged commit c9f791f into unicode-org:main Jul 12, 2026
8 checks passed

		@@ -0,0 +1,195 @@
		<?xml version="1.0" encoding="UTF-8" standalone="no"?>

Uh oh!

Uh oh!

Conversation

grhoten commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grhoten left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grhoten Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nciric left a comment

Choose a reason for hiding this comment

Uh oh!

grhoten commented Jul 8, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grhoten Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nciric commented Jul 8, 2026

Uh oh!

grhoten commented Jul 8, 2026

Uh oh!

grhoten commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

grhoten commented Jun 27, 2026 •

edited

Loading

grhoten left a comment •

edited

Loading

grhoten Jun 27, 2026 •

edited

Loading

grhoten Jul 8, 2026 •

edited

Loading

grhoten commented Jul 8, 2026 •

edited

Loading