Skip to content

[Studio] Add real LLM gateway - #651

Open
Aias00 wants to merge 14 commits into
apache:rocketmq-studiofrom
Aias00:fix/studio-real-llm-gateway
Open

[Studio] Add real LLM gateway#651
Aias00 wants to merge 14 commits into
apache:rocketmq-studiofrom
Aias00:fix/studio-real-llm-gateway

Conversation

@Aias00

@Aias00 Aias00 commented Jul 28, 2026

Copy link
Copy Markdown

Summary

  • add an OpenAI-compatible LLM client for openai/deepseek/tongyi/ollama chat completions
  • add a primary LLM gateway that uses the real provider when config is complete and falls back to the stub only when config is incomplete
  • preserve the existing SSE contract by emitting message events with {"text": ...} and done events with [DONE]
  • add structured LLM failure metadata (code, hint, statusCode) for gateway errors and config-test failures
  • map LLM gateway failures through the global exception handler instead of falling back to generic 500 errors
  • surface config-test failure hints and machine-readable codes in the LLM settings UI
  • validate provider base URLs, token limits, and temperature before saving/testing LLM settings
  • normalize pasted OpenAI-compatible /chat/completions endpoints back to the provider base URL before requests are sent
  • load model options from the configured OpenAI-compatible provider /models endpoint, with built-in fallback models when the provider is unavailable
  • use saved LLM provider/model configuration on the AI chat page instead of hard-coded model choices

Scope

Failure handling covered

  • missing provider API key returns and displays llm.config.missing_api_key with a remediation hint
  • invalid provider base URL returns llm.config.invalid_api_base
  • out-of-range maxTokens returns llm.config.invalid_max_tokens
  • out-of-range temperature returns llm.config.invalid_temperature
  • unsupported configured providers return llm.config.unsupported_provider
  • upstream HTTP errors preserve upstream status and message where available
  • provider timeout returns 504 / llm.provider.timeout
  • empty provider response returns llm.provider.empty_response
  • non-JSON provider response returns llm.provider.malformed_response
  • provider model listing uses GET {apiBase}/models and falls back to built-in provider models on timeout, upstream error, or malformed model responses
  • frontend API client passes through domain payloads that use string code fields, while retaining numeric Result envelope handling

Non-goals

  • no tool calling in this slice
  • no multi-turn history changes
  • no Azure/Bedrock native adapters yet

Tests

  • Latest AI page wiring: 447883a [Studio] Use LLM config on AI chat page
  • Latest backend provider-model enhancement: 8dfea98 [Studio] Load LLM models from provider
  • JAVA_HOME=$(/usr/libexec/java_home -v 21) mvn -q -Dtest=OpenAiCompatibleLlmClientTest,LlmConfigServiceTest,OpenAiCompatibleLlmGatewayTest,LlmControllerTest,GlobalExceptionHandlerTest test
  • JAVA_HOME=$(/usr/libexec/java_home -v 21) mvn -q test
  • npm test -- --run src/api/llm.test.ts src/api/client.test.ts src/pages/studio/__tests__/LlmSettings.test.ts
  • npm run build
  • npm run lint
  • git diff --check

Copilot AI review requested due to automatic review settings July 28, 2026 01:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Aias00

Aias00 commented Jul 28, 2026

Copy link
Copy Markdown
Author
image image image image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants