[Engine] OutputParser prematurely terminates stream on prose words matching tool prefix heuristics

Problem:
In DeepSeek-V4.1 with reasoning disabled (reasoning_effort: "none"), prompts requesting log analysis with fields like "severidad, tecnica MITRE y recomendacion" trigger a premature termination (finish_reason: "stop") at token 19 or right at ---\n\n## 3. Recom.
Conversely, when asking the model to replace the term with "pasos de mitigacion" or when prompting for unrelated headings like ## Recuperación del Sistema, generation completes fully but finishes with finish_reason: "length" while emitting the raw <｜end▁of▁sentence｜> token as plaintext.

Environment:

Strata Version: 0.2.6 (standalone server/serve)
OS: Linux (Ubuntu 24.04 LTS / SLES)
GPU: 2x NVIDIA L20 (48 GB VRAM each)
Model / Quant: DeepSeek-V4.1-Flash (deepseek-2.00bpw)
Settings: --max-context 65536 --gpus 0,1, gateway with serve/server.py and serve/frontend.py

Reproduce:
Execute the following curl request without declaring any tools:
curl -s http://127.0.0.1:18000/v1/chat/completions
-H "Content-Type: application/json"
-H "Authorization: Bearer <API_KEY>"
-d '{
"model": "DeepSeek-V4.1-Flash",
"messages": [
{
"role": "user",
"content": "Analiza este log: 2026-10-25T03:14:22Z sshd[4821]: Failed password for invalid user admin from 185.220.101.47 port 55221 ssh2. Da severidad, tecnica MITRE y recomendacion."
}
],
"reasoning_effort": "none",
"max_tokens": 2048,
"temperature": 0.2
}'

Expected:
The model generates the full response (~500 tokens) and halts cleanly upon reaching the EOS token without emitting raw delimiter text.

Actual:
The stream halts abruptly at completion token 19 with finish_reason: "stop":
{
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "# Análisis del evento de seguridad\n\n## 1. Datos del evento\n\n"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 62,
"completion_tokens": 19,
"total_tokens": 81
}
}

Evidence:

When substituting "recomendacion" with "pasos de mitigacion" on the exact same log prompt, the stream runs completely (479 tokens, ~50.1 tok/s) across multiple Markdown levels (##, ###).
Prompts asking for ## Recuperación del Sistema also complete without halting, but both successful completions finish with finish_reason: "length" and end with the literal glyph <｜end▁of▁sentence｜>.
Inspection in serve/server.py shows that self.stop_ids does not include the exact token ID corresponding to <｜end▁of▁sentence｜> (with the U+2581 glyph), while OutputParser in serve/frontend.py enters an early stop state on specific output prefixes when parsing chat completions without declared tools.

Reproduction status:
reproduced locally

Thanks for the issue. To keep the tracker readable, every item needs an area prefix in the title and the parts of the issue form filled in. This one is missing:

Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other], then a short description. Example: "[Engine] decode stops after 2 minutes".

Edit the issue (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment /reopen.

Details: CONTRIBUTING.md

Thanks, this follows the format now.

Hello, may I ask what the output speed would be for the L20（48G） x 2 deployment of deepseek v4.1 flash?

6. Hello, may I ask what the output speed would be for the L20（48G） x 2 deployment of deepseek v4.1 flash?

Hello! Under this deployment setup, you can expect an output generation speed of approximately 45 to 50 tokens per second during the decoding phase. Key Deployment Insights & Hardware Details

Inference Performance:
    Output Speed: ~45–50 tokens/sec (decoding phase).
    Quantization: Highly compressed 2.00 bpw quantization (deepseek-2.00bpw), which keeps memory footprint low while maintaining standard model architecture execution.
    Context Window: Up to 65,536 tokens supported.

GPU & Memory Layout:
    Hardware: 2× NVIDIA L20 GPUs (48 GB VRAM each, ~90 GB combined total).
    KV Cache Strategy: fp4 precision with active streaming—65,536 positions per layer kept in VRAM, while offloading excess state to system RAM.
    Expert Offloading: mmap-experts: off with a 64 GB SSD cache enabled to mitigate I/O bottlenecks when swapping weights.

Kubernetes & System Resources:
    Resource Caps: Capped at 48 CPUs and 192 GiB RAM via K8s container limits (sufficient for the host offloading requirements).
    CPU Execution: Configured with --cpu-threads 32, leaving headroom under your 48 CPU limit for Kubernetes overhead and background I/O tasks.
    Capabilities: IPC_LOCK and SYS_RESOURCE enabled, preventing memory page swapping and permitting real-time resource priority locks required by the inference runtime (vds4-0.1).

Performance & Compatibility Notes

I feel like something might be missing or suboptimal in this setup. For comparison, with Qwen 3.8 Flash Next, I achieve 180 to 200 tokens per second, with peaks reaching up to 270 tokens per second. Additionally, while Qwen works seamlessly with the Web GUI, this DeepSeek deployment currently only functions through direct API calls and fails to work properly with the Web UI interface.

The engine is fine; the gateway config is the cause. Without "arch": "deepseek41" the server gives DeepSeek the Qwen stop ids (<|im_end|>, <|endoftext|>). DeepSeek's tokenizer cuts those strings into pieces, so a | or > in a reply ends it, and the real end token (id 1) is not a stop id, so it prints as text. On two A6000 with your request: 730 to 1161 tokens with the arch, 192 to 436 without it.

Until the next engine, add "arch": "deepseek41" to the gateway config. Fixed on our side; the fix ships in the next update (0.2.8). With no arch named, the server now reads the pack's tokenizer and serves it as deepseek41. strata launch sets the arch, so Terminal and app users do not hit this.

Server: a config that names no "arch" but ships DeepSeek's tokenizer …

Fixed in 0.2.8: with no arch in the gateway config, the server reads the pack's tokenizer and serves it as deepseek41, and the real end token now ends the reply. strata launch already sets the arch. Run strata update. Closing.

Merge upstream v0.2.8 into memory-core: the Arc-pair and four-card ro…

7. > 6. 您好，请问部署两块 L20（48G）x 2 的 deepseek v4.1 闪存的输出速度是多少？

<img alt="图像" width="1283" height="817" src="https://private-user-images.githubusercontent.com/18372225/688340111-3a7a48a1-96c9-43da-ad63-98e0b11ebd52.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3OTMzMjEwMTAsIm5iZiI6MTc5MzMyMDcxMCwicGF0aCI6Ii8xODM3MjIyNS82ODgzNDAxMTEtM2E3YTQ4YTEtOTZjOS00M2RhLWFkNjMtOThlMGIxMWViZDUyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjEwMzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYxMDMwVDAwMzgzMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg2MWMxMGMxN2MwOGZjN2EwMDdmMjZjOGFhMmZmYzQ3MzllNDFiMzA3NzE5MTVjYTZlNjNjNzVhNjk2YWM3M2ImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JnJlc3BvbnNlLWNvbnRlbnQtdHlwZT1pbWFnZSUyRnBuZyJ9.3r28gEWexBxScspv3hbnF9dRGOde0Mr3Sh2bfxfZwKk">
您好！在此部署配置下，解码阶段的输出生成速度预计约为每秒 45 至 50 个令牌。关键部署信息和硬件详情
```
Inference Performance:
    Output Speed: ~45–50 tokens/sec (decoding phase).
    Quantization: Highly compressed 2.00 bpw quantization (deepseek-2.00bpw), which keeps memory footprint low while maintaining standard model architecture execution.
    Context Window: Up to 65,536 tokens supported.

GPU & Memory Layout:
    Hardware: 2× NVIDIA L20 GPUs (48 GB VRAM each, ~90 GB combined total).
    KV Cache Strategy: fp4 precision with active streaming—65,536 positions per layer kept in VRAM, while offloading excess state to system RAM.
    Expert Offloading: mmap-experts: off with a 64 GB SSD cache enabled to mitigate I/O bottlenecks when swapping weights.

Kubernetes & System Resources:
    Resource Caps: Capped at 48 CPUs and 192 GiB RAM via K8s container limits (sufficient for the host offloading requirements).
    CPU Execution: Configured with --cpu-threads 32, leaving headroom under your 48 CPU limit for Kubernetes overhead and background I/O tasks.
    Capabilities: IPC_LOCK and SYS_RESOURCE enabled, preventing memory page swapping and permitting real-time resource priority locks required by the inference runtime (vds4-0.1).
```


    
      
    

      
    

    
  
性能与兼容性说明
我感觉这个配置可能存在一些问题，或者并非最佳方案。作为对比，使用 Qwen 3.8 Flash Next 时，我能达到每秒 180 到 200 个令牌，峰值甚至可达每秒 270 个令牌。此外，虽然 Qwen 可以与 Web GUI 无缝协作，但目前这个 DeepSeek 部署只能通过直接 API 调用运行，无法与 Web UI 界面正常工作。

Thank you very much for your reply. It is very useful to me.

Navigation Menu

Repository navigation

[Engine] OutputParser prematurely terminates stream on prose words matching tool prefix heuristics #4137

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Engine] OutputParser prematurely terminates stream on prose words matching tool prefix heuristics #4137

Description

Activity

github-actions commented on Oct 25, 2026

github-actions commented on Oct 25, 2026

aniya105 commented on Oct 29, 2026

jlhs5 commented on Oct 29, 2026

Niko1221 commented on Oct 29, 2026

Niko1221 commented on Oct 29, 2026

aniya105 commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions