sachin-prabhakar · Oct 28, 2026 · Oct 28, 2026 · Oct 28, 2026 · Oct 28, 2026 · Oct 28, 2026
@@ -70,6 +70,7 @@ Important settings to review before deployment:
 - `llmApiGateway.metrics.enabled` to expose a metrics port on the Service and Deployment (default: `false`)
 - `llmApiGateway.metrics.serviceMonitor.enabled` to create a Prometheus `ServiceMonitor` (requires `metrics.enabled`)
 - `llmApiGateway.config.lastCluster.*` for session last-cluster hints sent to the LLM Request Router (default: disabled)
+- `llmApiGateway.config.decisionImageTokens` for the flat input token estimate of each image on `/v1/decisions` and `/v1/systemone` (default: `1024`)
 - `llmApiGateway.config.exposeRouterErrorCodes` to return the LLM Request Router rejection reason in `error.code` of OpenAI-format error responses (default: `false`)
 - `llmApiGateway.olric.*` for embedded rate-limit and last-cluster state and peer discovery. The Olric node starts whenever `olric.enabled` is `true`, even with rate limiting off
 - `llmApiGateway.vault.*` for JWT authentication path, role, and audience values used by the Vault Agent injector

@@ -32,6 +32,7 @@ data:
   STARGATE_LAST_CLUSTER_LOOKUP_TIMEOUT: {{ $cfg.lastCluster.lookupTimeout | quote }}
   STARGATE_LAST_CLUSTER_LOCAL_MAX_ENTRIES: {{ $cfg.lastCluster.localMaxEntries | int64 | toString | quote }}
   NVCF_GATEWAY_INFERENCE_WRITE_TIMEOUT: {{ $cfg.inferenceWriteTimeout | quote }}
+  NVCF_DECISION_IMAGE_TOKENS: {{ $cfg.decisionImageTokens | int64 | toString | quote }}
   NVCF_GRPC_ADDR: {{ $cfg.nvcfGrpcAddr | quote }}
   NVCF_GRPC_INSECURE: {{ $cfg.nvcfGrpcInsecure | toString | quote }}
   NVCF_GRPC_TIMEOUT: {{ $cfg.nvcfGrpcTimeout | quote }}

@@ -70,6 +70,10 @@ llmApiGateway:
     # reading. Applies only while a write is in progress, so it does not cap
     # total response time or upstream pauses. 0s disables the bound.
     inferenceWriteTimeout: 60s
+    # Flat input token estimate for each image on /v1/decisions and
+    # /v1/systemone. The image payload is never counted by size. Must be 0 or
+    # more; 0 counts images as free.
+    decisionImageTokens: 1024
     nvcfGrpcAddr: "api.nvcf.svc.cluster.local:9090"
     nvcfGrpcInsecure: false
     nvcfGrpcTimeout: 2s

@@ -36,11 +36,14 @@ for kv in ENABLED=false TTL=10m LOOKUP_TIMEOUT=20ms LOCAL_MAX_ENTRIES=100000; do
   test "$(read_config "$default_manifest" "STARGATE_LAST_CLUSTER_${kv%%=*}")" = "${kv#*=}" ||
     fail "STARGATE_LAST_CLUSTER_${kv%%=*} default must be ${kv#*=}"
 done
+test "$(read_config "$default_manifest" NVCF_DECISION_IMAGE_TOKENS)" = 1024 ||
+  fail "NVCF_DECISION_IMAGE_TOKENS default must be 1024"
 
 helm template llm-api-gateway "$chart_dir" \
   --namespace nvcf \
   --set-string llmApiGateway.image.repository=example.invalid/llm-api-gateway \
   --set llmApiGateway.config.nvcfGrpcInsecure=true \
+  --set llmApiGateway.config.decisionImageTokens=2048 \
   --set llmApiGateway.metrics.enabled=true \
   --set llmApiGateway.metrics.serviceMonitor.enabled=true \
   --set llmApiGateway.config.lastCluster.enabled=true \
@@ -49,6 +52,8 @@ helm template llm-api-gateway "$chart_dir" \
 
 test "$(read_config "$override_manifest" NVCF_GRPC_INSECURE)" = true ||
   fail "explicit plaintext transport override did not reach the ConfigMap"
+test "$(read_config "$override_manifest" NVCF_DECISION_IMAGE_TOKENS)" = 2048 ||
+  fail "decisionImageTokens override must render as NVCF_DECISION_IMAGE_TOKENS"
 # 1000000 checks that a large cap renders as a plain integer.
 for kv in ENABLED=true LOCAL_MAX_ENTRIES=1000000; do
   test "$(read_config "$override_manifest" "STARGATE_LAST_CLUSTER_${kv%%=*}")" = "${kv#*=}" ||

@@ -106,14 +106,18 @@ or the pod is killed mid-drain.
 
 ## Route mapping
 
-`vanityGateway.mappingConfig.v2config` has three sections:
+`vanityGateway.mappingConfig.v2config` has four sections:
 
 - `openai`: per-endpoint model routes, keyed by endpoint (`chatCompletions`,
-  `completions`, `embeddings`, `responses`, and the image endpoints). Each route
-  requires `modelName` and `functionID`.
+  `completions`, `embeddings`, `responses`, `decisions`, and the image
+  endpoints). Each route requires `modelName` and `functionID`. `decisions`
+  serves the beta `POST /v1/decisions` route.
 - `anthropicMessages`: native Anthropic Messages model routes, served on
   `POST /v1/messages`. Routes use the same fields as `openai.chatCompletions`
   and share the API host set by `openai.host`.
+- `typesafeSystemOne`: beta TypeSafe System One model routes, served on
+  `POST /v1/systemone`. A top-level sibling of `anthropicMessages`, with the
+  same fields as `openai.decisions`.
 - `vanity`: host-based routes, each requiring a `host` and a `paths` map. Each
   path requires `path` and `functionID`.
 
@@ -123,13 +127,14 @@ across all routes.
 
 A model may set `functionType: LLM` to be served by the LLM Gateway instead of
 the invocation service, supported in `openai.chatCompletions`,
-`openai.responses`, `openai.embeddings`, and `anthropicMessages`. Callers still send the public `modelName`; the gateway
+`openai.responses`, `openai.embeddings`, `openai.decisions`, `anthropicMessages`, and
+`typesafeSystemOne`. Callers still send the public `modelName`; the gateway
 rewrites the request model to `functionID/modelName` before forwarding.
 
 The values schema enforces what the gateway checks at startup, so a values file
 that would fail the container fails the render instead:
 
-- `functionType` is rejected outside those four endpoints
+- `functionType` is rejected outside those endpoints
 - `usePexec`, `outgoingPathOverride`, and `sessionTimeout` are rejected on such
   a model, since the LLM Gateway ignores them. An explicit `false`, `""`, or `0`
   is accepted, because that is what an absent key produces
@@ -162,6 +167,43 @@ Messages. Upgrade Pylon workers, the Stargate router, and the LLM Gateway
 before enabling Messages routes. See `docs/overview/llm-gateway.md` for the
 request, accounting, and Claude Code details.
 
+Decision models (beta) are mapped per route. A model is enabled for a route only
+when listed in that route's section, and a function that serves both formats is
+listed in both:
+
+```yaml
+vanityGateway:
+  mappingConfig:
+    v2config:
+      openai:
+        host: api.example.com
+        decisions:
+          example_decision-model:
+            modelName: example/decision-model
+            functionID: <function-id>
+            functionVersionID: <function-version-id>
+          example_decision-model-llm:
+            modelName: example/decision-model-llm
+            functionID: <llm-function-id>
+            functionType: LLM
+      typesafeSystemOne:
+        example_decision-model:
+          modelName: example/decision-model
+          functionID: <function-id>
+          functionVersionID: <function-version-id>
+```
+
+Request and response bodies pass through unchanged, and responses are not
+streamed. With `usePexec: true` the invocation service normally sends the request
+to the default invocation path registered for the function, whichever route the
+client called, so a `usePexec: true` function serves only its default path.
+A function that serves both routes should set `usePexec: false`. Models with
+`functionType: LLM` need LLM Gateway and Stargate builds that serve these routes;
+enable them only after those builds run in the environment. Request body
+sampling (`promptCacheKey`, `firstMessageHash`) and `promptCacheKeyHeaders` are
+rejected in the decision sections. See the service README for error shapes and
+`/v1/models` behavior.
+
 Shadow traffic is supported. Each shadow target is resolved from the same model
 table and routed by its own `functionType`, so a shadow of an LLM model reaches
 the LLM Gateway, and an LLM model may shadow a model served by the invocation

@@ -145,6 +145,9 @@
                 "anthropicMessages": {
                   "$ref": "#/definitions/openAIStickyEndpointRoutes"
                 },
+                "typesafeSystemOne": {
+                  "$ref": "#/definitions/openAILLMEndpointRoutes"
+                },
                 "vanity": {
                   "$ref": "#/definitions/vanityRoutes"
                 }
@@ -273,6 +276,29 @@
                                 "required": [
                                   "embeddings"
                                 ]
+                              },
+                              {
+                                "properties": {
+                                  "decisions": {
+                                    "not": {
+                                      "additionalProperties": {
+                                        "not": {
+                                          "properties": {
+                                            "functionType": {
+                                              "const": "LLM"
+                                            }
+                                          },
+                                          "required": [
+                                            "functionType"
+                                          ]
+                                        }
+                                      }
+                                    }
+                                  }
+                                },
+                                "required": [
+                                  "decisions"
+                                ]
                               }
                             ]
                           }
@@ -303,6 +329,29 @@
                         "required": [
                           "anthropicMessages"
                         ]
+                      },
+                      {
+                        "properties": {
+                          "typesafeSystemOne": {
+                            "not": {
+                              "additionalProperties": {
+                                "not": {
+                                  "properties": {
+                                    "functionType": {
+                                      "const": "LLM"
+                                    }
+                                  },
+                                  "required": [
+                                    "functionType"
+                                  ]
+                                }
+                              }
+                            }
+                          }
+                        },
+                        "required": [
+                          "typesafeSystemOne"
+                        ]
                       }
                     ]
                   }
@@ -357,6 +406,9 @@
         "embeddings": {
           "$ref": "#/definitions/openAILLMEndpointRoutes"
         },
+        "decisions": {
+          "$ref": "#/definitions/openAILLMEndpointRoutes"
+        },
         "responses": {
           "$ref": "#/definitions/openAIStickyEndpointRoutes"
         },

@@ -76,11 +76,21 @@ assert_renders "a model without functionType renders with an empty endpoint"
 write_values "${ENDPOINT}" chatCompletions "            functionType: llmGateway"
 assert_rejected "value must be 'LLM'" "an unrecognized functionType is rejected"
 
-for section in responses embeddings; do
+for section in responses embeddings decisions; do
   write_values "${ENDPOINT}" "${section}" "${LLM}"
   assert_renders "an LLM model renders in ${section}"
 done
 
+write_values "" decisions "${LLM}"
+assert_rejected "/vanityGateway/config/llmGatewayEndpoint" "an LLM decisions model requires an endpoint"
+
+write_values "${ENDPOINT}" decisions "${LLM}
+            usePexec: true"
+assert_rejected "a_model/usePexec" "usePexec is rejected on an LLM decisions model"
+
+write_values "" decisions ""
+assert_renders "a decisions model without functionType renders with an empty endpoint"
+
 for section in completions imageGenerations imageEdits imageVariations; do
   write_values "${ENDPOINT}" "${section}" "${LLM}"
   assert_rejected "openai/${section}/a_model" "functionType is rejected in ${section}"
@@ -156,6 +166,46 @@ write_messages_values "${ENDPOINT}" "${MESSAGES_LLM}
           sessionTimeout: 900"
 assert_rejected "anthropicMessages/a_model/sessionTimeout" "sessionTimeout is rejected on an LLM Messages model"
 
+# typesafeSystemOne is a top-level sibling of anthropicMessages.
+write_system_one_values() {
+  local endpoint="$1" extra="$2"
+  cat > "${VALUES}" <<EOF
+vanityGateway:
+  image:
+    registry: example.com
+    repository: foo/bar
+  config:
+    llmGatewayEndpoint: "${endpoint}"
+  mappingConfig:
+    v2config:
+      openai:
+        host: api.example.com
+      typesafeSystemOne:
+        a_model:
+          modelName: acme/a-model
+          functionID: 00000000-0000-0000-0000-000000000001
+${extra}
+EOF
+}
+
+write_system_one_values "" "${MESSAGES_LLM}"
+assert_rejected "/vanityGateway/config/llmGatewayEndpoint" "an LLM System One model requires an endpoint"
+
+write_system_one_values "${ENDPOINT}" "${MESSAGES_LLM}"
+assert_renders "an LLM System One model with an endpoint renders"
+
+write_system_one_values "" ""
+assert_renders "a System One model without functionType renders with an empty endpoint"
+
+write_system_one_values "${ENDPOINT}" "${MESSAGES_LLM}
+          usePexec: true"
+assert_rejected "typesafeSystemOne/a_model/usePexec" "usePexec is rejected on an LLM System One model"
+
+write_system_one_values "${ENDPOINT}" "${MESSAGES_LLM}
+          customHeaders:
+            X-Priority: \"5\""
+assert_rejected "typesafeSystemOne/a_model/customHeaders" "an X-Priority custom header is rejected on an LLM System One model"
+
 # The proxy preserves the caller path and the transport only speaks http(s).
 for bad in "ftp://llm-gateway:8080" "http://llm-gateway:8080/v1"; do
   write_values "${bad}" chatCompletions "${LLM}"

@@ -47,9 +47,14 @@ ${extra}
 EOF
 }
 
-# anthropicMessages sits beside openai, so its entries are two spaces shallower.
+# anthropicMessages and typesafeSystemOne sit beside openai, so their entries
+# are two spaces shallower.
 write_anthropic_values() {
-  local extra="$1"
+  write_toplevel_values anthropicMessages "$1"
+}
+
+write_toplevel_values() {
+  local section="$1" extra="$2"
   cat > "${VALUES}" <<EOF
 vanityGateway:
   image:
@@ -59,7 +64,7 @@ vanityGateway:
     llmGatewayEndpoint: "${ENDPOINT}"
   mappingConfig:
     v2config:
-      anthropicMessages:
+      ${section}:
         a_model:
           modelName: acme/a-model
           functionID: 00000000-0000-0000-0000-000000000001
@@ -247,7 +252,7 @@ write_values chatCompletions "            shadowModelName: acme/a-model-next
               - perBearerKey"
 assert_rejected "shadowSamplingMethod" "a legacy method after random is rejected"
 
-for section in completions embeddings imageGenerations; do
+for section in completions embeddings imageGenerations decisions; do
   write_values "${section}" "            shadows:
               - modelName: acme/a-model-next
                 samplingMethod: promptCacheKey"
@@ -262,6 +267,26 @@ for section in completions embeddings imageGenerations; do
   assert_rejected "promptCacheKeyHeaders" "promptCacheKeyHeaders is rejected in ${section}"
 done
 
+write_toplevel_values typesafeSystemOne "            shadows:
+              - modelName: acme/a-model-next
+                samplingMethod: promptCacheKey"
+assert_rejected "samplingMethod" "promptCacheKey is rejected in typesafeSystemOne"
+
+write_toplevel_values typesafeSystemOne "            shadowModelName: acme/a-model-next
+            shadowSamplingMethod: firstMessageHash"
+assert_rejected "shadowSamplingMethod" "legacy firstMessageHash is rejected in typesafeSystemOne"
+
+write_toplevel_values typesafeSystemOne "            promptCacheKeyHeaders:
+              - x-session-id"
+assert_rejected "promptCacheKeyHeaders" "promptCacheKeyHeaders is rejected in typesafeSystemOne"
+
+write_toplevel_values typesafeSystemOne "            shadows:
+              - modelName: acme/a-model-next
+                samplingMethod:
+                  - perBearerKey
+                  - random"
+assert_renders "ordered bearer sampling renders in typesafeSystemOne"
+
 write_anthropic_values "            promptCacheKeyHeaders:
               - x-session-id
             shadows:

@@ -12,18 +12,21 @@ emit legacy service-prefixed metric names.
 
 Use bounded labels only. Do not add request IDs, session IDs, function IDs,
 organization IDs, project IDs, raw URLs, raw prompts, authorization values, or
-other unbounded request fields as metric labels.
+other unbounded request fields as metric labels. The one exception is
+`function_id`, which the HTTP request metrics and the decision usage metric
+carry.
 
 ## Metrics
 
 | Metric name | Type | Source endpoint | Labels | Notes |
 | --- | --- | --- | --- | --- |
-| `llm_api_gateway_http_requests_total` | Counter | `llm-api-gateway:9464/metrics` | `method`, `route`, `status` | Total inbound HTTP requests. `route` is the templated route, not the raw URL path. |
+| `llm_api_gateway_http_requests_total` | Counter | `llm-api-gateway:9464/metrics` | `method`, `route`, `status`, `function_id` | Total inbound HTTP requests. `route` is the templated route, not the raw URL path. For `/v1/decisions` and `/v1/systemone`, zero series with `function_id="none"` exist before traffic (request statuses 200, 400, 401, 404, 429, 500, 502, 504). |
 | `llm_api_gateway_http_request_duration_seconds` | Histogram | `llm-api-gateway:9464/metrics` | `method`, `route`, `status` | Inbound HTTP request duration in seconds. |
-| `llm_api_gateway_http_active_requests` | Gauge | `llm-api-gateway:9464/metrics` | `method`, `route` | Current in-flight inbound HTTP requests. |
+| `llm_api_gateway_http_active_requests` | Gauge | `llm-api-gateway:9464/metrics` | `method`, `route`, `function_id` | Current in-flight inbound HTTP requests. For `/v1/decisions` and `/v1/systemone`, zero series with `function_id="none"` exist before traffic. |
 | `llm_api_gateway_upstream_requests_total` | Counter | `llm-api-gateway:9464/metrics` | `upstream`, `result`, `status` | Total outbound upstream requests. `upstream` is a bounded service name such as `llm-request-router`. |
 | `llm_api_gateway_upstream_request_duration_seconds` | Histogram | `llm-api-gateway:9464/metrics` | `upstream`, `result`, `status` | Outbound upstream request duration in seconds. |
 | `llm_api_gateway_llm_tokens_total` | Counter | `llm-api-gateway:9464/metrics` | `endpoint`, `token_type`, `stream` | LLM token counts reported by upstream providers. `token_type` is a bounded enum such as `prompt`, `completion`, or `total`. |
+| `llm_api_gateway_decision_usage_observations_total` | Counter | `llm-api-gateway:9464/metrics` | `endpoint`, `status`, `function_id` | Responses on `/v1/decisions` and `/v1/systemone` by whether they reported `usage.input_tokens`. `status` is `observed` or `absent`; encoded responses, non-2xx responses and 2xx responses with no body are not counted. Series for both routes with `function_id="none"` start at zero. |
 | `llm_api_gateway_provider_time_seconds` | Histogram | `llm-api-gateway:9464/metrics` | `endpoint`, `phase`, `stream` | Provider-reported timing phases in seconds. |
 | `llm_api_gateway_stream_first_token_seconds` | Histogram | `llm-api-gateway:9464/metrics` | `endpoint` | Time from stream request start to first token in seconds. |
 | `llm_api_gateway_stream_duration_seconds` | Histogram | `llm-api-gateway:9464/metrics` | `endpoint`, `status` | Total stream duration in seconds. |