feat: add Russian-language support to the corrections signal (#6)#28
Merged
Conversation
The corrections keyword catalog was English-only, so Russian
course-corrections (нет, не туда, стоп, отмена, …) went undetected.
Refactor src/adapter/correction.ts from hardcoded English token arrays into
an additive per-language LANG_CATALOGS {en, ru} table; adding a language is
now a catalog-only change (matching logic is language-agnostic). A shared
normalize() — lowercase, ё→е, curly→straight apostrophe, punctuation→space,
whitespace collapse — is applied to both input and tokens, so «нет,» and нет
match identically. Word-boundary check upgraded to Unicode \p{L} so нет no
longer matches нетворк; English behavior is preserved exactly. Emits only
{matched, shape} — privacy contract intact.
Tests: RU coverage across all four shapes, affirmations (да/отлично/точно),
normalization (guillemets, ё→е cross-form), word-boundary anti-false-positives,
cross-language bleed, shape precedence; plus a pipeline-level Russian
prose-leak assertion in test/privacy.test.ts with a load-bearing
corrections>=1 check that the text was actually processed.
Docs: EN+RU note in CONSUMER_GUIDE FAQ, DESIGN §4.2, ARCHITECTURE module row.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6.
Why
The
correctionssignal's keyword catalog (src/adapter/correction.ts) was English-only, so Russian course-corrections (нет,не туда,стоп,отмена, …) went undetected — they never set thecorrectionflag, under-counting friction on any EN/RU bilingual session.DESIGN.md§4.2 already listed«не туда»as a correction example.What changed
LANG_CATALOGS: { en, ru }, mapped to the same fourshapevalues. Per-shape token lists are merged + normalized + deduped once at module load; the matching loop is language-agnostic. Adding a language is now a catalog-only change — no matching-logic edits. Latin/Cyrillic are disjoint scripts, so the merge cannot introduce cross-language bleed.ё→е, curly/smart quotes → straight apostrophe, every other non-letter/digit/apostrophe → space, whitespace collapse. So«нет,»≡нет,ещё≡еще, and hyphenatedпо-другомуunifies with its two-word form.startsWithTokennow uses\p{L}for continuation, soнетno longer matchesнетворк/неточный— and the prior English semantics (apostrophe + digit = boundary) are preserved exactly.{matched, shape}; raw text is consumed and discarded.Tests
test/correction.test.ts: RU coverage across all four shapes; affirmations (да/отлично/точно) do not match; normalization («нет,»,стоп!,ё→еcross-form); word-boundary anti-false-positives (нетворк/неточный); cross-language bleed; shape-precedence lock (по-другому→ redirect).test/privacy.test.ts§(f): pipeline-level Russian prose-leak assertion across human/json/calibrate + warnings, with a load-bearingcorrections >= 1check proving the text was actually ingested/detected (so non-leak isn't vacuously true).correction.test.tscases stay green.Docs
EN+RU note added to
CONSUMER_GUIDE.md(FAQ),DESIGN.md§4.2, and theARCHITECTURE.mdmodule-table row.Verification
npm run build✓ ·tsc --noEmit✓ · 423 tests pass.Review
Reviewed via
requesting-code-review(3 parallel reviewers: detector core / tests+privacy / docs). No Critical. Applied:неверноone-word synonym gap, vacuousё→еtest fix, privacy §(f) load-bearing assertion, misleadingдаtest rationale, deadпо-другомуduplicate removal, precedence-comment correction, positive-precedence + cross-bleed test strengthening, "stripped"→"ignored" wording. Accepted as-is: bareno,length-floor edge, short-token substring FP risk (consistent with pre-existing English), micro-allocation.🤖 Generated with Claude Code