Skip to content

Initial implementation#1

Draft
imnasnainaec wants to merge 34 commits into
mainfrom
develop
Draft

Initial implementation#1
imnasnainaec wants to merge 34 commits into
mainfrom
develop

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

README

https://github.com/sillsdev/python-sil-lift/tree/develop#sil-lift

Devin review

https://app.devin.ai/review/sillsdev/python-sil-lift/pull/1

Summary as of e5947db

Initial implementation of sil-lift, a Python library for LIFT (Lexicon Interchange Format) 0.13. The library provides:

  • Full object model: 35+ typed dataclasses for all LIFT 0.13 elements
  • Byte-fidelity passthrough: load() → save() with no edits is byte-identical; untouched entries keep their exact source bytes even when siblings change
  • Streaming: open_reader() / open_writer() for O(one-entry) memory on large files
  • Validation: RELAX NG + authored ranges schema + semantic checks (duplicate-guid, dangling-ref, etc.)
  • Canonical sort: LiftSorter-informed, locale-independent
  • CLI: validate / stats / sort / check-media / export
  • LIFT-folder handling: .lift-ranges companion discovery, media_refs() / missing_media()

Key architectural decisions:

  • All out-of-schema content (unknown elements, attributes, comments) is carried losslessly in Extras buckets
  • The passthrough layer uses a raw-byte scanner (_scan.py) to locate entry spans, then SHA-256 digests to detect changes
  • The streaming API yields the same Entry types as full-document mode
  • Validation is always explicit (never on load/save); the schema layer masks href attributes and groups children by tag to work around libxml2 limitations with real FLEx output

This change is Reviewable

imnasnainaec and others added 20 commits July 17, 2026 10:04
@imnasnainaec @claude
Project scaffolding (pyproject, src layout, configs, dormant CI, docs…
b955eed 
… skeleton)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Vendor lift-0.13.rng + assemble test corpus with provenance
3149084 
Sources pinned by SHA in PROVENANCE.md files. Corpus bytes are
normalization-exempt via .gitattributes (fidelity tests depend on it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Corpus tooling (0.12->0.13 migration, synthetic generator) + corpus s…
59556ad 
…anity tests

Migration validated against the vendored RNG; four fixtures are documented as
inherently RNG-invalid (lang-less etymology forms; FLEx file://C:/ range hrefs
that fail libxml2's anyURI check - see PROVENANCE.md, affects design).
All local checks green: ruff, ruff format, mypy --strict, 74 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Text model + full-document reader
2766bb4 
All 35 RNG elements as slotted dataclasses (quirks honored: two field
variants, form-shaped gloss, three-way extensibility split). Parser never
rejects well-formed 0.13: out-of-schema content lands in per-node opaque
Extras residue (unknown attrs/elements, comments, PIs, stray text, malformed
typed attributes). Version guard names the version. 128 tests, all checks
green (ruff, mypy --strict).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Writer + losslessness
c464f44 
Canonical serializer (custom indenter that never touches mixed content -
lxml pretty_print corrupts <text> whitespace, caught on the Sango corpus);
byte-span scanner for top-level entry/header spans; unchanged
documents save byte-identically, untouched entries stay verbatim under edits,
touched entries re-serialize canonically with residue re-emitted at clamped
anchors. Hypothesis round-trip properties (caught the empty-fragment
<text></text> vs <text/> asymmetry); no-lxml-leakage public API test;
fidelity contract documented (README, docs/en/fidelity.md). 187 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Ranges + folder handling
f28e63f 
RangesFile with the same passthrough fidelity as .lift (per-range verbatim
reuse); companion discovery (sibling + href-basename fallback for FLEx's
dangling file://C:/ hrefs); save() writes companions together; all_ranges()
merged view; media_refs()/missing_media() handling backslash hrefs with
spaces (WeSay). Authored lift-ranges-0.13.rng from the vendored grammar's
defines - validates the hand-crafted pair; FLEx files fail it (trait/field
inside range-element, third documented FLEx deviation, carried in Extras).
200 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Validation module
dcb1862 
Problem stream with file/entry/line addressing; RNG layer with two deliberate
deviations from raw libxml2 (documented in _validate.py): href masking (anyURI
false-flags every FLEx file; sil-lift reports uri-not-rfc warnings instead)
and tag-grouped validation (libxml2 rejects legal interleave once an element
type appears in >1 run - minimal repro: 'f n f n' invalid, 'f f n n' valid).
Ranges-schema layer over companions; semantic checks: duplicate-guid,
dangling-ref (entry id/guid/sense id targets), range-parent integrity,
undefined-range-value (NFC-normalized: FLEx writes .lift in NFC but
.lift-ranges in NFD), duplicate-form-lang (the RNG's Schematron rule),
missing-media. Finds real defects in the Sango export (2 dangling range
parents, 1 undefined POS value). Fail-fast wrappers raise on first error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Negative-test corpus + validator test suite
c8521b1 
Eight hand-authored fixtures, each documenting its defect and expected
Problem; tests verify addressing (entry id/guid/line), warning-vs-error
levels, fail-fast behavior, clean-corpus cleanliness, C# parity case
(duplicate-guid), and the real Sango defects (2 dangling range parents,
1 undefined POS). 263 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Canonical sort
eb07a33 
Native implementation mirroring C# LiftSorter's rules (consulted at
libpalaso 4840de8): entries by case-insensitive guid (extended to (guid,id)
casefold so guidless entries sort deterministically), header ranges/
range-elements by id, field definitions by tag, senses in file order,
text/span whitespace untouched. Lexicon.sort()/RangesFile.sort() in-place;
canonicalize(src, dst) for diff-ready fully-canonical output. Acceptance:
idempotent, byte-deterministic, sorted output RNG-valid + semantically
equal, sort+save keeps untouched entry bytes verbatim. 275 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Streaming read/write
b051705 
LiftReader (iterparse; header parsed eagerly, entries lazy, clear()/sibling
cleanup internal, version guard at open) and LiftWriter (canonical chunk per
entry; output byte-identical to canonical_document by construction;
exception leaves the file visibly unterminated). Same Entry types as full
mode; full-vs-streaming model equality verified over the whole corpus;
stream-copy of Sango preserves models; bounded memory measured: 400k-entry
/ ~340MB file stream-copied with <100MB working-set growth (40k-entry
variant in the default suite). Extras anchor index excluded from equality
(position hint, not content). 313 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
CLI demo, documentation site, release polish
b638102 
sil-lift CLI (validate/stats/sort/check-media; stdlib argparse; exit codes
0/1/2) exercising every scope pillar, verified against the corpus incl. the
filename-with-space fixture. Docs: task-oriented guides (read/edit/write,
validate, large files, folder+media, CLI), fidelity page, mkdocstrings API
reference; mkdocs build --strict green. Wheel built and verified in a fresh
Python 3.12 venv (import, load, CLI). 323 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
docs: add 'Differences from the C# libraries' page
42d22b3 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
CLI export subcommand + bulk-edit worked example docs
6273ecb 
export: LIFT -> CSV/TSV, one row per leaf sense (a sense with subsenses is
a grouping node), streaming with two-pass language auto-detection or
--langs; 6 new tests. Docs: guides/bulk-edit-glosses.md with a runnable,
verified script (load -> edit glosses incl. subsenses -> validate-gate ->
save) and the per-entry fidelity story; nav/index wired. 329 tests green;
mkdocs build --strict green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
CI: split into build.yml and test.yml
f59ee8a 
Least-privilege workflow permissions (contents: read); one action per step,
every step named; lint and format checks as separate steps; build matrix on
the oldest and newest supported Pythons (3.11, 3.14). Supply-chain pinning:
actions by commit SHA (checkout v4.3.1, setup-python v5.6.0), build/twine
by version (1.5.0 / 6.2.0).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Pin ruff and mypy exactly in the dev extra; tidy pyproject
708f711 
New ruff/mypy releases change behavior and break CI on innocent code, so
they are pinned and bumped deliberately; the test tools keep floating
floors so a breaking pytest/hypothesis/lxml release surfaces early. Also:
lxml-stubs floor (>=0.5), alphabetized dev extra / docs extra / ruff
select, condensed comments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Add .editorconfig with byte-exact carve-outs; restyle YAML configs
8281ecb 
.editorconfig: UTF-8/LF/final-newline defaults, 2-space YAML/TOML, and a
do-nothing carve-out for corpus fixtures and vendored schemas (mirrors
.gitattributes) so editors can't corrupt byte-exact test payloads.
mkdocs.yml/crowdin.yml: dedented sequence style; plugin order now i18n
first, search last (indexes localized content) - the arrangement
mkdocs-static-i18n recommends.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Docs and editor-config batch
4529147 
Docs tree: long-line prose (Crowdin-friendly, per new editorconfig rule),
underscore emphasis, realigned tables; index.md guide list dropped (nav
covers it). CONTRIBUTING.md added (dev setup, byte-exact corpus rules,
mkdocs workflow) with a README pointer. mkdocs.yml: footer social links.
.editorconfig: md indent groups, docs-tree overrides (4-space list indent
for python-markdown, no wrap guidance), plus .vscode markdown ruler.
Root markdown wrapped at 80. Code fences verified byte-identical; strict
docs build and full check suite green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Document deferred maintenance triggers: Python 3.11 floor, MkDocs->Ze…
6c5b256 
…nsical

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@imnasnainaec @claude
Record created Crowdin project in crowdin.yml dormancy note
7937bce 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Agent docs, branch/release policy, PyPI release workflow; version 0.1.0
d9c858f 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec imnasnainaec self-assigned this Jul 17, 2026
imnasnainaec and others added 8 commits July 17, 2026 11:31
@imnasnainaec @claude
Docs on GitHub Pages; PyPI/project URLs in README, docs, pyproject; C…
1918042 
…rowdin sync note

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Validate what save() would write, not the loaded bytes
b339981 
Review findings (PR #1): iter_problems on a loaded-then-edited lexicon
validated the original source bytes, hiding problems introduced by edits
and mis-addressing semantic findings once entries were added, removed, or
sorted. Render through the writer instead — byte-identical to the source
for untouched documents, so file line numbers still hold.

Also: check-media now skips remote/absolute hrefs via _normalize_href
(mirrors missing_media); save() docstring covers same-directory renames;
large-files.md notes root attributes are not carried in streaming.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Exclude tests/ from the sdist; resolve the corpus license-review flag
634bf30 
The sdist shipped all of tests/corpus, redistributing fixtures whose
upstream repos have no license (lift-standard, LiftTweaker) or an
AGPL-by-reference one (flashgrab) in the PyPI artifact. PROVENANCE.md now
records the review outcome and that corpus files are not covered by the
repository's MIT license.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Harden two review-noted edges: re-key ranges_files on save; tolerate …
b320e04 
…None lines

Review round 2 (PR #1): after a relocate-save the ranges_files dict stayed
keyed by the companions' pre-save paths (latent — nothing reads keys after
save today); _nearest_entry stopped scanning at an entry without a
sourceline instead of skipping it (unreachable on lxml-parsed input).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec
Trim test summaries; Set 90% coverage floor
fee1e30
@imnasnainaec
Remove phase/research/decision/milestone refs
96a5035
@imnasnainaec @claude
Bump actions/checkout to v7.0.0, actions/setup-python to v6.3.0
3cecd8b 
Both were several majors behind (v4->v7, v5->v6). Checked release notes:
neither breaks plain checkout/setup-python usage with no special inputs,
provided the runner is >= v2.327.1 -- true unconditionally for GitHub-hosted
runners. upload-pages-artifact, deploy-pages, and gh-action-pypi-publish
were already on latest.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@imnasnainaec @claude
Fix 3 validation coverage gaps; scheme-valid dummy GUIDs in tests
e5947db 
Review findings: stats undercounted media (missed variant.pronunciations,
media_refs already covered it); dangling-ref missed refs nested in
variant.relations; duplicate-form-lang only checked 3 of ~14 Multitext
fields (replaced with a generic dataclass-tree walk, which also catches
Multitext nested inside Annotation.content -- unbounded, not just the
missed fields).

Also: dummy GUIDs in test fixtures were structurally invalid UUIDs (wrong
version/variant nibbles). Repeating-digit blocks are kept for at-a-glance
obviousness, with -4444-8888- inserted at the version/variant positions --
valid-by-scheme, distinguishable from real data by any grep.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
imnasnainaec and others added 6 commits July 20, 2026 13:46
@imnasnainaec @claude
Fix audit findings: doc/comment consistency, test enforcement, writer…
82e4fe6 
… blank line

Docs and comments brought in line with the code:
- README lists the export subcommand; docs build uses `mkdocs build --strict`
- schemas/PROVENANCE.md: lift-ranges-0.13.rng is authored, not "planned"
- cli.md stats sample shows the real sense count (4541); export = "leaf sense"
- validate.md snippet defines `lex` before using it
- csharp-differences.md: ranges validated as companions, no standalone entry
- _cli.py drops the stale "decision-document section D" reference
- test_stream.py drops the stale "implementation plan" reference
- check.py docstring accurate; CONTRIBUTING documents the 90% coverage floor
- _canonical.canonicalize docstring notes companions are not rewritten
- _model.Lexicon.load docstring: candidates are all tried, not a "fallback"
- _model find/save and RangesFile.save docstrings document their ValueError

Behavior and robustness:
- check.py runs pytest under --cov=sil_lift --cov-fail-under=90 (CI parity)
- test_public_api no longer silently skips members whose annotations use
  TYPE_CHECKING-only names; it checks raw annotations for etree/lxml instead
- _writer._slot_bytes: touched entries/headers/ranges no longer emit a blank
  line in the byte-passthrough path
- _validate parses internally-rendered bytes with a hardened parser
  (resolve_entities=False, no_network=True)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@imnasnainaec @claude
Fix canonical-output gluing, header wrapper residue, and duplicate he…
e55c88b 
…aders

- Canonical serialization (canonical_document / canonical_ranges_document) now
  guarantees a newline separator between chunks, so byte-reused (untouched)
  spans no longer glue to their neighbors when a structural edit
  (add/remove/reorder) routes through the canonical fallback. Pure-canonical
  output is unchanged, since its chunks already end in a newline.
- Header gains ranges_extra / fields_extra: out-of-schema residue on the
  <ranges>/<fields> wrapper elements (unknown attributes, stray nodes) now
  stays on the wrapper instead of migrating up onto <header> during canonical
  re-serialization. The reader test is updated to assert the new attribution.
- The duplicate-<header> guard tracks whether a header has been seen instead of
  testing Header truthiness, so an empty first <header/> is treated as the
  model header and a second <header> is preserved as residue rather than
  silently overwriting it.
- AGENTS.md documents that the hard-wrapped root docs backslash-escape bare
  Markdown list markers (e.g. `sort \+ save`) so a reflow cannot turn them into
  stray bullets, and that docs/en/ (not hard-wrapped) needs no such escaping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@imnasnainaec @claude
Add JSON, strict, and no-check-media options to CLI validate
fe65d6f 
`sil-lift validate` now accepts:

- `--format json`: emit findings as a single JSON object (a problems array
  plus an errors/warnings summary) instead of human text, for machine
  consumption.
- `--strict`: treat warnings as errors, exiting 1 on any warning.
- `--no-check-media`: skip the filesystem media-presence check, suppressing
  missing-media findings (noise when validating a freshly generated export
  whose media lives elsewhere).

Document the flags and a JSON example in the CLI guide, note that validate's
exit codes and JSON schema are a supported, SemVer-covered interface, and drop
the "demo" framing from the CLI's description.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@imnasnainaec @claude
Note in AGENTS.md that commit messages must be self-contained
0065b08 
Commit messages should describe the diff and stand on their own, without
referencing conversation, tasks, audits, or review context.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@imnasnainaec @claude
Add LIFT-export tooling: builders, validator checks, CLI, container
605b200 
Support producing and validating LIFT exports.

Construction API for building LIFT from scratch:
- `Lexicon.add_ranges_file()` attaches a `.lift-ranges` companion and adds the
  header `<range href>` references; `save()` writes a path-less companion
  beside the target `.lift`.
- `RangesFile.add_range()` and `Range.add_element()` builder helpers.
- `open_writer(..., ranges=...)` emits the companion on the streaming path.

Validation:
- `dangling-ranges-href` warning when a relative header `range/@href` resolves
  to no companion (absolute/file:// FLEx-style hrefs are exempt).
- `missing-id` errors for entries without a guid or senses without an id,
  opt-in via `iter_problems(require_ids=True)`.

CLI:
- `validate --require-ids`, `validate -` (read from stdin), and
  `stats --format json`.

Container and GitHub Action wrapping `sil-lift validate` (`Dockerfile`,
`action.yml`, `docker-entrypoint.sh`, container build workflow) so a non-Python
CI pipeline can run the conformance check with no local Python toolchain.

Docs: a "Producing conformant LIFT" guide covering the conformance gate, the
`.lift-ranges` companion shape, and the multitext and XML-escaping rules.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@imnasnainaec @claude
Pin shell scripts to LF line endings
8ac02cc 
`* text=auto` lets a Windows checkout rewrite `docker-entrypoint.sh` with CRLF,
whose `\r` shebang fails in the Linux container. Force LF for `*.sh`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@imnasnainaec imnasnainaec

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@imnasnainaec