Introduce --receptor flag to optimize test output for LLMs and AI Agents (e.g., --receptor=llm)

[dprada]

What's the problem this feature will solve?

With the massive adoption of terminal-based AI Agents (such as Claude Code, Codex CLI, Aider, and custom LLM-driven TDD loops), pytest is increasingly being executed by non-human actors.

Currently, agents parse stdout or rely on heavy workarounds. Standard human-readable layouts (verbosity, ANSI colors, repeated source code blocks in tracebacks, progress bars) consume an immense amount of context tokens unnecessarily, increasing API costs and execution latency.

Conversely, using aggressive existing flags like --tb=line -q drops crucial failure context (such as complex multi-line dictionary or object diffs), forcing the agent to run blind, hallucinate fixes, and fall into loops. Relying only on process exit codes (0/1) tells the agent THAT something failed, but not WHAT failed, requiring extra tool calls to read secondary logs.

Describe the solution you'd like

We propose introducing a native --receptor (recipient/sink) architecture to decouple the consumer profile from the framework's core reporting logic. Instead of just a cosmetics-based --output-format, a --receptor flag tells pytest WHO is digesting the data, allowing the framework to optimize for token density, semantic structure, and context windows.

Example command:
pytest --receptor=llm

Behavior and Technical Specifications by Receptor

To protect backward compatibility, --receptor=human remains the default, preserving 100% of the current pytest behavior. The framework will switch its rendering heuristics completely when --receptor=llm is active:

• On Success (Green Suite): Completely suppress test lists, headers, and environment blocks. Output a single, atomic, token-efficient string: OK: {count} passed in {time}s.
• On Failure (Red Suite):
  • Strip all visual text-decorations (no ====, no horizontal lines, no progress percentages, no ANSI colors).
  • No source code redundancy: Do not print the surrounding lines of code where the exception occurred (the AI agent already has the source files in its context window or can inspect them independently).
  • High semantic density: Provide only the file path, line number, exception type, and the exact raw comparison diff wrapped in clean, lightweight semantic delimiters (like minified XML tags) that maximize Transformer attention efficiency.

Proposed Taxonomy

• --receptor=human (Default: Rich terminal UI, tracebacks, progress bars)
• --receptor=llm (Ultra-dense, zero-redundancy, token-optimized for AI)
• --receptor=machine / ci (Optimized for standard CI/CD logging systems without interactive noise)

Alternative Solutions

We have considered using pytest-json-report or custom shell wrappers (pytest -q --tb=line | grep ...). However, these approaches either blind the agent to complex assert diffs or introduce heavy token overhead by forcing the agent to ingest entire JSON structures back into its prompt context.

While plugins can dump JSON or custom logs, forcing every AI CLI developer or team to maintain custom intermediary wrappers to clean stdout is inefficient, fragmented, and error-prone. A native, zero-config --receptor=llm would establish pytest as the gold standard for AI-native test runners.

Additional context

Phased Rollout Strategy (Starting as a Community Plugin)

We understand and respect pytest's strict core-minimization philosophy. Therefore, this architecture doesn't necessarily need to land in the core codebase on day one.

We propose a phased approach:

1. Phase 1 (Proof of Concept): Implement this design as an external plugin (pytest-receptor or pytest-llm-output) to refine the --receptor=llm token-saving heuristics and gather real-world telemetry from AI CLI users.
2. Phase 2 (Core Hook/Flag Adoption): Once stabilized and widely adopted by the community/AI tools, evaluate upstreaming the --receptor flag or providing a native hook so that third-party plugins can register custom receptors smoothly without hacking the TerminalReporter.

Implementation Strategy (How it could work underneath)

Instead of adding heavy formatting logic to the core or relying on string post-processing, pytest could decouple its terminal reporter hook (_pytest.terminal.TerminalReporter).

• The Receptor Hook Interface: When pytest_runtest_logreport fires, the framework queries the active receptor strategy.
• Short-circuiting the Traceback Generator: Under --receptor=llm, the traceback formatting bypasses _pytest._code.code.ReprTraceback's heavy layout builder. It directly extracts the raw ExceptionInfo attributes (err.type, err.value, and the exact failing frame location) and serializes them in a compact format.
• Token Attention Optimization: Failure reports can be wrapped in plain text markup tags (like <test_failure>...</test_failure>). Modern Transformers are natively trained on massive datasets containing HTML/XML; they process these boundaries with significantly lower attention-mechanism noise than arbitrary lines of dashes (-----).

[dprada]

Follow-up: we built it, and some of what I proposed above turned out to be wrong.

Following the phased approach in the original post, we implemented this as an external plugin — uibcdf/pytest-receptor — and have been running it against a real 8,000-test scientific suite. Reporting back, because three of the claims I made above did not survive contact with an implementation, and because we hit some pytest internals that may be of interest independently of whether --receptor is ever a core concern.

What the measurements look like. Against pytest -q -n 12 on 8,000 tests, cl100k_base: a fully green run goes from 812 tokens to 24; one broken fixture failing 200 tests goes from 25,481 to 114; six unrelated bugs from 1,497 to 285. The green figure is worth a note — -q still prints one progress character per test, so a successful 8,000-test run costs 812 tokens of dots before anything is wrong. The cascade figure is the one that matters: the value is not shorter formatting, it is that one root cause is printed once instead of two hundred times.

What I got wrong.

Minified XML. I claimed transformers process XML boundaries with "significantly lower attention-mechanism noise". I have no evidence for that, and I should not have asserted it. XML also costs more tokens than plain text for identical content, which works against the stated goal. We removed it. Escaping it correctly was a real source of malformed output while it lasted.

OK: {count} passed in {time}s. This specification is a bug, and we shipped it. Deriving success from the absence of failure reports means an empty run (exit 5) and an interrupted run (exit 2) both render as OK: 0 passed. Any outcome line has to come from pytest.ExitCode, and it should state the exit code so a consumer can check.

Short-circuiting the traceback generator. The suggestion to bypass ReprTraceback and read raw ExceptionInfo attributes does not work the way I assumed — see the first point below.

Three things about pytest internals we ran into.

1. --tb affects how longrepr is built, not only how it is displayed, so a plugin cannot suppress pytest's traceback output without also destroying the data it wanted to summarise. Filed separately as --tb affects longrepr construction, so plugins cannot suppress traceback display without losing traceback data #14720.
2. --no-summary gates the entire pytest_terminal_summary hook. A plugin setting it in order to quieten pytest also silences every third-party plugin that writes there — it swallowed pytest-cov's report for us until we noticed.
3. summary_exit_codes in terminal.py does not include ExitCode.INTERNAL_ERROR, so pytest_terminal_summary never fires on exit 3 — arguably the moment a consumer most needs to be told what happened.

On the proposal itself: I no longer think a --receptor flag belongs in core. An external plugin is sufficient, and consumer profiles in the core codebase would cut against pytest's minimisation philosophy. I am happy for this issue to be closed.

But "sufficient" deserves an honest qualifier, because it is the actual argument for the two API points above.

Collection is entirely public: pytest_runtest_logreport, pytest_collectreport, pytest_warning_recorded, pytest_sessionfinish. Suppression is not. To stop pytest rendering its own output while keeping the underlying data, the plugin currently has to:

• empty TerminalReporter.reportchars directly, an undocumented instance attribute, because --no-summary would silence every other plugin as well (point 2);
• set config.option.tbstyle = "no" inside pytest_sessionfinish — after every longrepr has been built, before the reporter renders. That works only because of the build/display coupling in --tb affects longrepr construction, so plugins cannot suppress traceback display without losing traceback data #14720. The workaround is the bug;
• import create_terminal_writer from _pytest.config and assign reporter._tw, for its token-accounting mode.

So a plugin can do this today, but only by depending on four undocumented mechanisms, one of which exploits a defect I have just reported. That, rather than a new flag, is what I think is worth the team's attention.