Skip to content

Add Japanese tokenizer (fugashi) and minimal unit test#223

Open
supernaiter wants to merge 1 commit into
microsoft:mainfrom
supernaiter:pr-japanese-tokenizer
Open

Add Japanese tokenizer (fugashi) and minimal unit test#223
supernaiter wants to merge 1 commit into
microsoft:mainfrom
supernaiter:pr-japanese-tokenizer

Add Japanese tokenizer (fugashi) and minimal unit test

953e5a0
Select commit
Loading
Failed to load commit list.
Microsoft GitHub Policy Service / license/cla succeeded Jul 12, 2025 in 0s

All CLA requirements met.

This check verifies that the author has agreed to a CLA with Microsoft.

View more details on Microsoft GitHub Policy Service