mojo-unicodedata

Unicode normalization (NFC/NFD/NFKC/NFKD) and case folding in pure Mojo, mirroring Python's unicodedata. No Python dependencies, no FFI.

As of mid-2026 the Mojo ecosystem has no library for Unicode normalization. mojo-unicodedata fills that gap: the four normalization forms and full case folding, code-generated from the Unicode Character Database and validated against the official conformance corpus. It mirrors Python's unicodedata where sensible, so normalize("NFC", s) and casefold(s) do what you'd expect coming from CPython.

Normalization is the un-glamorous prerequisite for correct text handling: comparing user input, deduplicating strings, building search indexes, or canonicalizing identifiers all break subtly when "café" (composed) and "café" (decomposed) are treated as different strings. This library makes them equal.

Coming from Python

If you know Python's unicodedata, the normalization entry points map directly:

Python	mojo-unicodedata
unicodedata.normalize("NFC", s)	normalize("NFC", s)
unicodedata.normalize("NFKD", s)	normalize("NFKD", s)
unicodedata.is_normalized("NFC", s)	is_normalized("NFC", s)
s.casefold()	casefold(s)

These are module-level functions. If you normalize many strings, construct a Normalizer() once and call .normalize(form, s) / .casefold(s) on it to reuse the loaded tables.

What it handles

• All four normalization forms: normalize("NFC"|"NFD"|"NFKC"|"NFKD", s), matching Python's unicodedata.normalize signature.
  • NFD / NFKD: canonical / compatibility decomposition, followed by canonical ordering of combining marks.
  • NFC / NFKC: decomposition and ordering, then canonical composition.
• Full case folding: casefold(s), matching Python's str.casefold (Unicode CaseFolding.txt common + full mappings). ß to ss, ff to ff, Σ to σ.
• is_normalized(form, s): whether a string is already in a given form.
• Hangul done right: syllables are decomposed and composed algorithmically (UAX #15 §16 SBase/LBase arithmetic), not tabled.
• Reusable Normalizer: the tables parse once into a Normalizer; reuse it for bulk work (the free functions build one per call for convenience).

Correctness

Passes 20,034 / 20,034 rows (100%) of the official Unicode NormalizationTest.txt conformance corpus: every part, including the character-by-character test, canonical order test, PRI #29 test, canonical closures, and chained primary composites. A row passes only when every invariant for all four forms holds.

@Part0 # Specific cases                          : 45 / 45
@Part1 # Character by character test             : 17086 / 17086
@Part2 # Canonical Order Test                    : 1936 / 1936
@Part3 # PRI #29 Test                            : 194 / 194
@Part4 # Canonical closures (excluding Hangul)   : 735 / 735
@Part5 # Chained primary composites              : 38 / 38
TOTAL: 20034 / 20034 rows passing

Run it yourself with pixi run conformance. Plus 23 unit tests (pixi run test) covering ASCII passthrough, composed/decomposed é both directions, singletons (Å, Ω), compatibility mappings, Hangul LV/LVT, canonical ordering, case folding, idempotence, and error handling.

What it deliberately does NOT do

• No character metadata. Names, general categories, numeric values, bidi classes: unicodedata.name(), category(), numeric() are out of scope for v0.1. Only the fields normalization and folding need are shipped.
• No NFC/NFD quick-check tables. is_normalized normalizes and compares rather than using the *_QC properties, so it's correct but not the fast path. A quick-check is a natural v0.2 addition.
• No locale/Turkic case folding. casefold uses the language-neutral full folding (statuses C+F); the Turkic T mappings are skipped, matching Python's str.casefold.

Install

With pixi:

pixi install
pixi run test
pixi run conformance

Or with uv:

uv venv
uv pip install mojo --index https://whl.modular.com/nightly/simple/ --prerelease allow
.venv/bin/mojo run -I src test/test_unicodedata.mojo

Requires a Mojo nightly (>=1.0.0b3).

Usage

from unicodedata import normalize, casefold, is_normalized, Normalizer

def main() raises:
    # Composed vs decomposed "café" compare equal after NFC.
    var a = String("café")                       # may be NFC or NFD on the wire
    var b = normalize("NFD", a)                   # e + combining acute
    print(normalize("NFC", a) == normalize("NFC", b))   # True

    print(casefold(String("Straße")))            # strasse
    print(normalize("NFKC", String("fi")))        # fi (ligature expanded)
    print(is_normalized("NFC", String("é")))     # True

    # Reuse a Normalizer for bulk work (tables parse once):
    var norm = Normalizer()
    for s in my_strings:
        _ = norm.normalize("NFC", s)

How the tables are built

The normalization and case-folding tables are code-generated from the Unicode Character Database, which is the whole trick to getting conformance right. scripts/generate_tables.py downloads (or reuses a local cache of) UnicodeData.txt, DerivedNormalizationProps.txt, and CaseFolding.txt, and emits src/unicodedata/tables.mojo as compact hex-encoded StaticString blobs (canonical + compatibility decompositions, non-zero combining classes, primary composition pairs minus the full composition exclusions, and full case foldings), parsed once at runtime into sorted arrays for binary search. It also regenerates the conformance fixture from NormalizationTest.txt.

To update to a new Unicode version (or regenerate from scratch):

python3 scripts/generate_tables.py

src/unicodedata/tables.mojo is generated; never hand-edit it. Edit the generator and re-run.

Tests

pixi run test
pixi run conformance

23 unit tests plus 20,034 / 20,034 official Unicode conformance rows, described above.

Part of a pure-Mojo library suite

Eleven pure-Mojo libraries that mirror familiar Python stdlib and PyPI APIs, filling gaps in the native Mojo ecosystem:

• mojo-xml — general-purpose XML parsing, an ElementTree-shaped DOM (Python's xml.etree.ElementTree)
• mojo-feed — RSS, Atom, and JSON Feed parsing (Python's feedparser)
• mojo-captions — SRT and WebVTT subtitle/transcript parsing (no Python stdlib parallel)
• mojo-html — HTML parsing and article extraction (Python's readability)
• mojo-markdown — CommonMark markdown parsing (Python's markdown)
• mojo-diff — text diffing (Python's difflib)
• mojo-template — a Jinja-flavored template engine (Python's jinja2)
• mojo-tar — tar archive reading and writing (Python's tarfile)
• mojo-redis — a Redis client (Python's redis-py)
• mojo-url — URL parsing and encoding (Python's urllib.parse)

Related Unicode work

• mzaks/mojo-unicode is a complementary testbed focused on Unicode case conversion (uppercase / lowercase) and UTF-8 utilities. It does not implement normalization, so the two libraries cover different halves of Unicode text handling; pair them if you need both case conversion and normalization.

Contributing

Issues and PRs welcome, especially strings that normalize or case-fold wrong (include the input as hex codepoints and, if comparing to Python, the unicodedata.unidata_version you used, since this library tracks the latest UCD, which may be newer than your local CPython). Run pixi run test and pixi run conformance before sending a PR, and regenerate tables.mojo via the generator rather than editing it by hand.

About

Built by Conor Bronsdon — host of Chain of Thought, a podcast about AI agents, infrastructure, and engineering. Find me on X or LinkedIn.

---

Disclaimer

All views, opinions, and statements expressed on this account/in this repo are solely my own and are made in my personal capacity. They do not reflect, and should not be construed as reflecting, the views, positions, or policies of Modular. This account is not affiliated with, authorized by, or endorsed by my employer in any way.

License

Licensed under the MIT License. The Unicode Character Database data used to generate the tables is subject to the Unicode License.