Skip to content

feat: add v2 BYOK OpenAI-compatible LLM credentials#217

Merged
suguanYang merged 6 commits into
mainfrom
feat/suguanYang/byok-llm-credentials
Jul 15, 2026
Merged

feat: add v2 BYOK OpenAI-compatible LLM credentials#217
suguanYang merged 6 commits into
mainfrom
feat/suguanYang/byok-llm-credentials

Conversation

@suguanYang

@suguanYang suguanYang commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add optional llm_config (OpenAI-compatible credentials) to v2 only (POST /v2/jobs, POST /v2/retrieval/query); v1 is unchanged.
  • Thread credentials through job metadata → worker parse (greenlet-scoped overrides) and agentic retrieval (async ContextVar).
  • Mask API keys in original_request snapshots; raw keys stay in job metadata for the worker only (MVP plaintext-at-rest).

Usage

Semantics

  • Flat root {api_key, model, base_url} → both text and vision
  • models.{text,vision} → same endpoint, different model ids (shared api_key / base_url)
  • text / vision objects → fully replace that channel (different endpoints / keys)
  • a channel with no resolved model keeps server defaults

One multimodal model

{
  "llm_config": {
    "api_key": "sk-...",
    "model": "gpt-4o",
    "base_url": "https://api.openai.com/v1"
  }
}

Same endpoint, different models

{
  "llm_config": {
    "api_key": "sk-...",
    "base_url": "https://api.openai.com/v1",
    "models": { "text": "gpt-4o-mini", "vision": "gpt-4o" }
  }
}

Two different provider endpoints

{
  "llm_config": {
    "text": {
      "api_key": "sk-...",
      "model": "gpt-4o-mini",
      "base_url": "https://api.openai.com/v1"
    },
    "vision": {
      "api_key": "sk-ali-...",
      "model": "qwen-vl-max",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    }
  }
}

Node SDK

import { Knowhere } from "@ontos-ai/knowhere-sdk";

const client = new Knowhere({ apiKey: "..." });

// Flat multimodal
await client.jobs.create({
  sourceType: "url",
  sourceUrl: "https://example.com/doc.pdf",
  llmConfig: {
    apiKey: "sk-...",
    model: "gpt-4o",
    baseUrl: "https://api.openai.com/v1",
  },
});

// Same endpoint, different models
await client.retrieval.query({
  query: "What is the refund policy?",
  useAgentic: true,
  llmConfig: {
    apiKey: "sk-...",
    baseUrl: "https://api.openai.com/v1",
    models: { text: "gpt-4o-mini", vision: "gpt-4o" },
  },
});

Companion PRs

Test plan

  • make check (ruff + pyright)
  • pytest packages/shared-python/shared/tests/test_llm_config.py
  • Flat llm_config applies to both channels
  • models map uses shared auth with different model ids
  • Split text + vision uses different endpoints
  • Confirm job responses do not expose raw API keys
  • Confirm v1 ignores / rejects llm_config

Allow users to pass text and vision provider configs on v2 job create
and retrieval so KNOWHERE can drive parsing and agentic retrieval with
caller-supplied OpenAI-compatible keys, models, and endpoints.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread packages/shared-python/shared/services/ai/llm_overrides.py Fixed
Return early when gevent is unavailable so CodeQL no longer flags a
bare ImportError pass in the BYOK override root lookup.

Co-authored-by: Cursor <cursoragent@cursor.com>
suguanYang and others added 4 commits July 15, 2026 21:38
Text-only or vision-only llm_config no longer cross-fills the other channel,
so text models cannot break VLM calls (and vice versa).

Co-authored-by: Cursor <cursoragent@cursor.com>
One shared provider slot covers both text and vision; text/vision still
override per channel when set.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use flat api_key/model/base_url for multimodal BYOK; keep text/vision
as full channel replacements for different provider endpoints.

Co-authored-by: Cursor <cursoragent@cursor.com>
Allow shared api_key/base_url with models.text / models.vision when only
the model id differs between channels.

Co-authored-by: Cursor <cursoragent@cursor.com>
@suguanYang
suguanYang merged commit 04f5adf into main Jul 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants