feat(perf): add workload_trace dataset plugin for production traffic replay

[qiumuyang]

Summary

Add a workload_trace dataset plugin that replays recorded production traffic with original timing, headers, and request bodies — enabling performance benchmarking against real-world workload patterns.

Closes #1489

Motivation

Existing perf datasets (openqa, share_gpt, random, etc.) generate synthetic traffic that may not reflect production load characteristics. Real production workloads have bursty arrival patterns, heterogeneous request shapes, and model-specific routing. This plugin bridges that gap by replaying captured JSONL traces verbatim.

Changes

New files

File	Description
evalscope/perf/plugin/datasets/workload_trace.py	Core plugin (249 lines)
evalscope/perf/utils/body_meta.py	Sideband key constants for per-request metadata
tests/perf/test_workload_trace.py	48 unit tests
tests/perf/test_workload_trace_e2e.py	7 E2E tests (mock HTTP server)

Modified files

File	What changed
arguments.py	--dataset-args CLI arg; --model / --number optional for trace datasets; requires_model validation
benchmark.py	Re-read total_count / warmup_count after plugin construction (plugins may adjust them); remove unused import numpy
plugin/api/base.py	extract_body_meta() — pops sideband keys before process_request; merges per-request headers with fill semantics (CLI headers win)
plugin/api/openai_api.py	Guard model=None when dataset carries model in body
plugin/datasets/base.py	provides_arrival_schedule / requires_model class attributes
plugin/datasets/__init__.py	Register import
core/strategies/open_loop.py	Trace-offset dispatch branch with deferred duration deadline for warmup
core/http_client.py	Call extract_body_meta before delegating to API plugin

Trace format

Each line in the JSONL file:

{"body": {"model": "qwen-plus", "messages": [...]}, "timestamp": 1700000000.0}
{"body": {"model": "qwen-max", "messages": [...]}, "timestamp": 1700000001.5, "headers": {"X-Tag": "exp"}, "request_id": "req-42", "completion_tokens": 256}

• body (dict or JSON string): complete request body, sent as-is.
• timestamp (number or ISO-8601): arrival time; only relative deltas matter.
• headers (optional): per-request HTTP headers (merged with fill semantics — CLI headers take precedence).
• request_id (optional): propagated to results for correlation.
• completion_tokens (optional): used by match_output_length.

Usage

# Basic replay
evalscope perf \
  --dataset workload_trace \
  --dataset-path trace.jsonl \
  --url http://localhost:8000/v1/chat/completions \
  --open-loop

# 2× speed, remap models, match recorded output lengths
evalscope perf \
  --dataset workload_trace \
  --dataset-path trace.jsonl \
  --url http://localhost:8000/v1/chat/completions \
  --open-loop \
  --dataset-args '{"speed": 2.0, "model_mapping": {"gpt-4": "qwen-max"}, "match_output_length": true}'

# Limit to first 500 requests
evalscope perf \
  --dataset workload_trace \
  --dataset-path trace.jsonl \
  --url http://localhost:8000/v1/chat/completions \
  --open-loop \
  --number 500

Dataset args (--dataset-args)

Key	Type	Default	Description
speed	float	1.0	Replay speed multiplier (2.0 = 2× faster)
model_override	str	None	Replace all models with this value
model_mapping	dict	None	Selectively remap model names
match_output_length	bool	false	Set max_tokens from completion_tokens and enable ignore_eos (requires vLLM or compatible)

Design decisions

• Sideband keys: per-request metadata (__evalscope_* prefixed) is embedded in the body dict and stripped by extract_body_meta before reaching API plugins. This avoids changing the plugin interface.
• Fill semantics for headers: {**trace_headers, **cli_headers} — CLI-set headers (e.g. Authorization from --api-key) always take precedence over trace headers, so explicit CLI configuration is never silently overridden by trace data.
• --model / --number optional: Trace data carries its own model and count. New requires_model class attribute on DatasetPluginBase controls validation.
• Deferred deadline: --duration countdown starts after warmup completes, so warmup doesn't consume benchmark time.

Testing

$ pytest tests/perf/test_workload_trace.py tests/perf/test_workload_trace_e2e.py -q
55 passed

Coverage includes: loading/validation, timestamp parsing (epoch + ISO-8601), body-as-string, model rewriting, speed scaling, warmup, --number truncation, match_output_length, header fill semantics, reserved key rejection, E2E body passthrough, and E2E warmup exclusion.

[gemini-code-assist]

Code Review

This pull request introduces the workload_trace dataset plugin, enabling the replay of recorded production traffic traces with their original arrival patterns in open-loop mode. It updates the argument parser, HTTP client, and open-loop strategy to handle per-request metadata, such as arrival offsets, custom headers, and request IDs. Feedback on the changes suggests several robustness and performance improvements, including replacing Pydantic models with plain dictionaries for trace records to reduce overhead, handling stringified epoch timestamps during normalization, preserving the list type of query_parameters.number, and adding defensive checks to prevent potential KeyError or AttributeError exceptions.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Update _compute_schedule to use dictionary access instead of object attribute access, matching the optimized dict-based record storage.

Suggested change

	def _compute_schedule(self) -> np.ndarray:
	if not self._records:
	return np.array([])
	t0 = self._records[0].timestamp
	return np.array([(r.timestamp - t0) / self._config.speed for r in self._records])
	def _compute_schedule(self) -> np.ndarray:
	if not self._records:
	return np.array([])
	t0 = self._records[0]['timestamp']
	return np.array([(r['timestamp'] - t0) / self._config.speed for r in self._records])

[gemini-code-assist]

Update build_messages to use dictionary access instead of object attribute access, matching the optimized dict-based record storage.

    def build_messages(self) -> Iterator[Dict[str, Any]]:
        match_output = self._config.match_output_length
        for i, record in enumerate(self._records):
            body = dict(record['body'])
            self._rewrite_model(body)

            completion_tokens = record.get('completion_tokens')
            if match_output and completion_tokens is not None:
                body['max_tokens'] = completion_tokens
                body['ignore_eos'] = True

            body[BODY_META_ARRIVAL_OFFSET] = float(self._schedule[i])
            if i < self._warmup_count:
                body[BODY_META_IS_WARMUP] = True
            headers = record.get('headers')
            if headers:
                body[BODY_META_HEADERS] = dict(headers)
            request_id = record.get('request_id')
            if request_id:
                body[BODY_META_REQUEST_ID] = request_id

            yield body

[gemini-code-assist]

Add a defensive check to ensure body is a dictionary before attempting to pop keys, preventing potential AttributeError if body is None or of an unexpected type.

        if not isinstance(body, dict):
            return headers, None
        extra_headers = body.pop(BODY_META_HEADERS, None)

[Copilot]

Pull request overview

Adds a new workload_trace perf dataset plugin to replay recorded JSONL production traffic with preserved request bodies and a trace-derived open-loop arrival schedule, enabling more realistic performance benchmarking than synthetic generators.

Changes:

• Introduces workload_trace dataset plugin with timestamp parsing, model rewriting options, warmup marking, and optional output-length matching.
• Adds “sideband” body-meta keys and a common extract_body_meta() path to strip them and apply per-request headers / request IDs during HTTP dispatch.
• Updates open-loop strategy to support trace-provided arrival offsets and defers --duration accounting until after warmup for trace-offset replay.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
evalscope/perf/plugin/datasets/workload_trace.py	Implements JSONL trace loading/validation, schedule computation, and message construction with body-meta sidebands.
evalscope/perf/utils/body_meta.py	Defines reserved __evalscope_* sideband keys used across the perf pipeline.
evalscope/perf/plugin/api/base.py	Adds extract_body_meta() to strip sideband keys and merge per-request headers; returns request correlation ID.
evalscope/perf/core/http_client.py	Invokes extract_body_meta() before request processing and propagates request IDs into result objects.
evalscope/perf/core/strategies/open_loop.py	Adds a dispatch path that uses trace-provided arrival offsets and handles warmup/deadline behavior for replay.
evalscope/perf/plugin/api/openai_api.py	Avoids writing model=None into the outgoing payload when the model is carried by the dataset body.
evalscope/perf/plugin/datasets/base.py	Adds dataset capability flags (provides_arrival_schedule, requires_model) for validation and scheduling behavior.
evalscope/perf/plugin/datasets/__init__.py	Registers the new dataset plugin import.
evalscope/perf/benchmark.py	Re-reads counts after dataset plugin construction so plugins can adjust number/warmup parameters.
evalscope/perf/arguments.py	Makes --model/--number optional at CLI level and validates them based on dataset capabilities; relaxes open-loop --rate validation for schedule-providing datasets.
tests/perf/test_workload_trace.py	Adds unit coverage for loading, rewriting, timestamp parsing, warmup marking, and match-output-length behavior.
tests/perf/test_workload_trace_e2e.py	Adds end-to-end tests with an in-process aiohttp server to validate timing and body/header propagation through the full pipeline.

[qiumuyang]

I think it's fine for --duration as a soft deadline, but happy to harden if needed.

[Yunnglin]

Pushed a few review-driven refinements on top of the latest commit:

1. --model no longer rewrites trace bodies (per #1489). The previous fallback silently turned --model X into model_override, collapsing multi-model traces into one. It now just logs a warning and each request keeps its own model; use --dataset-args model_override/model_mapping for explicit rewriting.
2. Guarded model=None in the remaining API plugins (dashscope / responses / embedding / rerank / custom). Only openai_api had the guard, so the others would write model: null when --model is omitted.
3. Gated per-request update_gpu_usage() to --api local. For remote APIs and local_vllm the client process holds no CUDA memory (measurement is always 0), and the first-request import torch (~3.5–6s) blocked the event loop — which distorted open-loop trace arrival timing for compact traces. Verified: a 3-request trace with 0.5s/0.8s gaps now replays within ~40ms instead of bursting.
4. Docs + test: added the workload_trace dataset mode, dataset-args, and usage examples (zh/en); added a test asserting --model does not rewrite the body.

E2E-verified against a mock OpenAI server: multi-model routing preserved, per-request headers applied, request_id propagated to the DB, and no __evalscope_* sideband keys leak into outgoing requests.