Skip to content

feat(eval): expose user_simulator_config in generate_responses#5733

Open
primenko-v wants to merge 7 commits into
google:mainfrom
primenko-v:propagate-user-simulator-config
Open

feat(eval): expose user_simulator_config in generate_responses#5733
primenko-v wants to merge 7 commits into
google:mainfrom
primenko-v:propagate-user-simulator-config

Conversation

@primenko-v
Copy link
Copy Markdown

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

N/A

2. Or, if no issue exists, describe the change:

Problem:
EvaluationGenerator.generate_responses constructs a UserSimulatorProvider() with no arguments, so the LLM-backed path always runs with the default BaseUserSimulatorConfig. There is no way for a caller to override the user-simulation model, max-allowed invocations, or custom instructions when driving multi-turn conversations through LlmBackedUserSimulator.

Solution:
Add an optional user_simulator_config parameter to generate_responses and forward it to UserSimulatorProvider(...). Callers can now pass an LlmBackedUserSimulatorConfig to customize the LLM-backed simulator.

The behavior is backward compatible:

  • When the argument is omitted, UserSimulatorProvider falls back to BaseUserSimulatorConfig() exactly as before.
  • Static eval cases are unaffected: the config is ignored by StaticUserSimulator.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

A unit test for the proposed change was added to tests/unittests/evaluation/test_evaluation_generator.py: TestGenerateResponses::test_generate_responses_forwards_llm_backed_user_simulator_config

All tests pass:

> uv run pytest tests/unittests/ -rs
...
================================== short test summary info ================================== 
SKIPPED [1] tests/unittests/integrations/crewai/test_crewai_tool.py:20: Requires Python 3.10+
================  5770 passed, 1 skipped, 2358 warnings in 129.40s (0:02:09) ================

The skipped test is not related to this change — it skips on main as well.

Manual End-to-End (E2E) Tests:

A reference setup lives at https://github.com/primenko-v/adk-x-mlflow (tag pr-demo/user-simulator-config).

It loads an LlmBackedUserSimulatorConfig from YAML and forwards it to EvaluationGenerator.generate_responses via the new user_simulator_config parameter — see src/mlflow_adk/simulate.py.

To reproduce (requires GOOGLE_CLOUD_PROJECT and ADC via gcloud auth application-default login):

git clone --recurse-submodules --branch pr-demo/user-simulator-config \
    https://github.com/primenko-v/adk-x-mlflow.git
cd adk-x-mlflow
cp .env.example .env  # fill in GOOGLE_CLOUD_PROJECT
uv sync
uv run python -m mlflow_adk.simulate --no-mlflow --output-traces traces.jsonl

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

@rohityan rohityan self-assigned this May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants