feat: add speculative decoding per-token latency metrics.

[phantomlei3]

Description

Add speculative decoding per-token latency metrics by amortizing each decode step’s TBT across the real tokens committed in that step. This improves latency observability for MTP/speculative decoding while excluding overlap placeholders and preempted tokens.

Original data
inter_token_latency_milliseconds_count : 127940
inter_token_latency_milliseconds_latency : 32
inter_token_latency_milliseconds_latency_9999 : 65
inter_token_latency_milliseconds_latency_cdf : "click to view"
inter_token_latency_milliseconds_latency_percentiles : "[33,34,35,56]"
inter_token_latency_milliseconds_max_latency : 65
inter_token_latency_milliseconds_qps : 890

added data when MTP K=1
speculative_per_token_latency_milliseconds_count : 127940
speculative_per_token_latency_milliseconds_latency : 18
speculative_per_token_latency_milliseconds_latency_9999 : 35
speculative_per_token_latency_milliseconds_latency_cdf : "click to view"
speculative_per_token_latency_milliseconds_latency_percentiles : "[17,31,33,35]"
speculative_per_token_latency_milliseconds_max_latency : 35
speculative_per_token_latency_milliseconds_qps : 890

Related Issues

Change Type

• Bug fix
• New feature
• Performance improvement
• Refactor
• Documentation
• Test
• Build or CI

Pull Request Checklist

Thank you for contributing to xLLM. Before requesting review, please make sure the following items are complete.

PR Title and Commit Messages

• The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Allowed types: feat, bugfix, docs, test, refactor, chore, style, revert, perf, model, build, release.
The subject should use clear English, start with a verb, include at least 4 words, and end with ..

Pre-commit Checks

• I have installed pre-commit by running pip install pre-commit or an equivalent command.
• I have installed the hooks with pre-commit install.
• I have run pre-commit run --all-files and fixed any reported issues.

If you are unsure how to set up pre-commit, see the pre-commit documentation.

Self Review

• I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
• I have rebased this PR onto the latest main branch.

Build and Test Coverage

• Tests have been added or updated as needed.
• CUDA: python setup.py build test has passed on a CUDA machine.
• NPU: python setup.py build test has passed on an NPU machine.
• MLU: python setup.py build test has passed on an MLU machine.

Reviewer Notes