feat: make transformers and the vLLM client optional dependencies (#31)

[hallerite]

Closes #31.

Why

transformers is a heavy dependency, and downstreams that keep training deps lightweight (e.g. TorchTitan/TorchTune, which load tokenizers via tokenizers) shouldn't have to pull it in just to use a renderer. This makes the heavy/engine-specific pieces opt-in, so the base install is lightweight and text-only renderers work with a bring-your-own tokenizer.

What changed

Tokenizer + Processor protocols (renderers/base.py) — structural types replacing transformers.PreTrainedTokenizer in every renderer's tokenizer/processor annotations. The module-level from transformers... import PreTrainedTokenizer is gone from all 13 renderer modules, so import renderers.<model> no longer drags in transformers.

transformers + fastokens → [transformers] extra. Needed only by the convenience helpers (load_tokenizer, create_renderer*), the offset-attribution fallback in attribute_text_segments, and the VLM renderers (image processors). _require_transformers() raises a clear pip install 'renderers[transformers]' error on those lazy paths when it's missing.

renderers.client → [vllm] extra. The vLLM /inference/v1/generate client is the only thing needing openai + httpx; it's no longer imported by renderers/__init__ (so import renderers stays free of HTTP/engine deps). OverlongPromptError is now imported from renderers.client (no top-level re-export).

Result — base pip install renderers core deps are just: numpy, tiktoken, jinja2, openai-harmony, prime-pydantic-config. Heavy bits are renderers[transformers] and renderers[vllm] (composable).

Caveats (documented in the README)

A bring-your-own tokenizer must satisfy the Tokenizer protocol (encode/decode/convert_tokens_to_ids/apply_chat_template + name_or_path/unk_token_id/eos_token_id), and per-token training attribution additionally needs tokenizer(..., return_offsets_mapping=True) — without it, attribution falls back to a vanilla HF tokenizer (the extra).

Tests

• New tests/test_no_transformers.py: subprocess-blocks transformers/fastokens/openai/httpx, then asserts import renderers + a text renderer's render/parse work, that no blocked module leaks into sys.modules, and that load_tokenizer errors with the install hint.
• Full suite green; ruff + ty clean.

🤖 Generated with Claude Code

---

Note

Medium Risk
Breaking for consumers that imported OverlongPromptError from renderers or assumed transformers/openai on base install; behavior is documented and guarded with new boundary tests.

Overview
This PR makes the base renderers install lightweight by moving heavy deps behind optional extras and letting text renderers run with a bring-your-own tokenizer.

Packaging: transformers and fastokens are no longer core dependencies; they install via renderers[transformers] (used by load_tokenizer / create_renderer*, offset attribution fallback, and VLMs). openai and httpx move to renderers[vllm] for renderers.client. Dev deps mirror both extras so CI still exercises those paths.

Typing / imports: New Tokenizer, ChatTemplateTokenizer, and Processor protocols in renderers/base.py replace PreTrainedTokenizer on all renderer modules, so importing renderers no longer pulls in transformers. VLMs load AutoProcessor through _require_transformers(), which raises a clear install hint if the extra is missing.

Public API: OverlongPromptError is dropped from renderers top-level exports; use from renderers.client import OverlongPromptError. Tokenizer, ChatTemplateTokenizer, and Processor are exported from the package root.

Tests / docs: tests/test_no_transformers.py subprocess-blocks optional deps and checks text render/parse, no leaked imports, and load_tokenizer errors. README documents extras, protocol expectations, and attribution caveats.

^{Reviewed by Cursor Bugbot for commit 5062105. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Make transformers and vLLM client optional dependencies in renderers

• Moves transformers, fastokens, openai, and httpx out of core dependencies into optional extras: renderers[transformers] and renderers[vllm].
• Introduces Tokenizer, ChatTemplateTokenizer, and Processor protocols in renderers/base.py so renderer classes no longer import from transformers directly.
• Adds a _require_transformers() helper that raises a clear ImportError with an install hint when transformers is not present and a code path needs it.
• import renderers no longer pulls openai/httpx into sys.modules; the vLLM client is only loaded on explicit import from renderers.client.
• Behavioral Change: OverlongPromptError is no longer exported as renderers.OverlongPromptError; callers must import it from renderers.client directly.

^{Macroscope summarized f19becd.}

[macroscopeapp]

Approvability

Verdict: Approved

Mechanical refactoring to make transformers and vLLM client optional dependencies. Type hints are changed from concrete types to protocols, and lazy imports provide clear error messages when extras are missing. Runtime behavior is unchanged when dependencies are installed.

^{You can customize Macroscope's approvability policy. Learn more.}

[hallerite]

not really satisfied with the way attribute_text_segments for character-offset attribution is handled, so further iterating.

[felipemello1]

hey @hallerite , thanks for writing this PR! I see that its "approved" (by a bot, so idk if that counts). I am just wondering if you think it will still get merged. Thank you!

[hallerite]

@felipemello1 I will try to get this in asap, probably tomorrow or the day after!

[felipemello1]

awesome, thanks @hallerite . Just to be clear: its not blocking us in TorchTitan. It is just a great feature. It is fine if you want to take your time to get it merged later. Ty!