From 02b06df359e1b5f8a428a49cc04f40ee5ed6f6b4 Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Thu, 9 Jul 2026 18:56:08 -0700 Subject: [PATCH 1/5] fix(helm): close blocker/real-gap findings from Helm chart best-practices audit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified every finding against the official Helm docs and Kubernetes Pod Security Standards docs before fixing, and validated each fix with helm lint/template plus the chart's own helm-unittest suite (65 -> 79 tests, all new tests confirmed to fail on the pre-fix code): - Blocker: values.schema.json documented "minimum 32/8 characters" on BETTER_AUTH_SECRET/ENCRYPTION_KEY/postgresql.auth.password but never enforced it. Added anyOf minLength-or-empty constraints (empty stays legal for existingSecret/ESO modes) — verified negative/positive cases live, no regression for any secret-delivery mode. - Real gap: copilot didn't support the External Secrets Operator mode the rest of the chart offers (app/postgresql/externalDatabase). Added external-secret-copilot.yaml, remoteRefs.copilot, and extended sim.copilot.validate with the same "map it or remove it" fail-fast guard app.env/realtime.env already have. Verified byte-identical rendering for the existing non-ESO path. - Real gap: the OpenTelemetry Collector was the only workload missing the shared Restricted-profile securityContext helpers (no container-level hardening at all). Wired sim.podSecurityContext/containerSecurityContext in, preserving the collector's original UID/GID/fsGroup. - Real gap: copilot templates hand-rolled label/selector blocks instead of using the chart's established sim..labels/selectorLabels pattern. Added sim.copilot.*/sim.copilotPostgresql.* helpers and refactored every consumer — confirmed byte-identical helm template output before/after (selector labels are immutable on upgrade, so this was verified, not assumed). - Documented (README): the ingressFrom default and readOnlyRootFilesystem posture, both real but intentional tradeoffs the audit flagged as underdocumented. Added extraVolumes/extraVolumeMounts to copilot's Deployment (realtime/pii already had it) so the readOnlyRootFilesystem guidance is actually actionable for all three stateless services. Deferred (nice-to-have, not blocking): pinning the two floating Postgres image tags, values.schema.json stubs for ~13 uncovered top-level sections, and an OTel collector image version bump — none are correctness issues. --- helm/sim/README.md | 21 +- .../sim/examples/values-external-secrets.yaml | 10 + helm/sim/templates/_helpers.tpl | 59 +++++- helm/sim/templates/deployment-copilot.yaml | 36 ++-- .../templates/external-secret-copilot.yaml | 38 ++++ helm/sim/templates/poddisruptionbudget.yaml | 7 +- helm/sim/templates/secrets-copilot.yaml | 8 +- .../statefulset-copilot-postgres.yaml | 31 +-- helm/sim/templates/telemetry.yaml | 7 +- helm/sim/tests/chart-computed-env_test.yaml | 10 +- helm/sim/tests/copilot-secret-modes_test.yaml | 189 ++++++++++++++++++ helm/sim/tests/env-defaults_test.yaml | 17 +- helm/sim/tests/helm-test-hook_test.yaml | 7 +- helm/sim/tests/networkpolicy_test.yaml | 6 +- helm/sim/tests/pdb-hpa_test.yaml | 7 +- helm/sim/tests/pii_test.yaml | 7 +- helm/sim/tests/pod-rollout_test.yaml | 9 +- helm/sim/tests/schema-secret-length_test.yaml | 59 ++++++ helm/sim/tests/secret-modes_test.yaml | 28 +-- helm/sim/tests/smoke_test.yaml | 2 +- .../telemetry-security-context_test.yaml | 53 +++++ helm/sim/tests/validators_test.yaml | 32 +-- helm/sim/values.schema.json | 12 +- helm/sim/values.yaml | 39 +++- 24 files changed, 568 insertions(+), 126 deletions(-) create mode 100644 helm/sim/templates/external-secret-copilot.yaml create mode 100644 helm/sim/tests/copilot-secret-modes_test.yaml create mode 100644 helm/sim/tests/schema-secret-length_test.yaml create mode 100644 helm/sim/tests/telemetry-security-context_test.yaml diff --git a/helm/sim/README.md b/helm/sim/README.md index 355aa5109c9..affa22fc2c7 100644 --- a/helm/sim/README.md +++ b/helm/sim/README.md @@ -219,7 +219,15 @@ Before installing in production, confirm each of the following: * **Secrets management** — provide secrets via External Secrets Operator (ESO) or pre-created Kubernetes Secrets. Never commit secrets to `values.yaml`. * **TLS / Ingress** — set the `cert-manager.io/cluster-issuer` annotation on the ingress and tune `proxy-body-size` / `proxy-read-timeout` for your workload. See commented examples in `values.yaml`. * **Network policy egress** — review `networkPolicy.egressExceptCidrs`. Defaults block cloud metadata endpoints (`169.254.169.254/32`, `169.254.170.2/32`); add your cluster's API server CIDR for stronger isolation. Custom egress rules go in `networkPolicy.egress` (a list). -* **Namespace hardening** — label the install namespace with Pod Security Standards `restricted` enforcement (`pod-security.kubernetes.io/enforce=restricted`). +* **Network policy ingress** — `networkPolicy.ingressFrom` defaults to `[{}]` (an empty peer selector), which allows ingress traffic from **any pod in the cluster**, not just your ingress controller. This is a deliberate simple default, not a locked-down one. On a shared or multi-tenant cluster, scope it down, e.g. to the ingress-nginx namespace: + ```yaml + networkPolicy: + ingressFrom: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + ``` +* **Namespace hardening** — label the install namespace with Pod Security Standards `restricted` enforcement (`pod-security.kubernetes.io/enforce=restricted`). All workloads set `runAsNonRoot`, drop all Linux capabilities, disable privilege escalation, and set `seccompProfile: RuntimeDefault` — the four controls the Restricted profile requires. `readOnlyRootFilesystem` is intentionally **not** defaulted anywhere (Postgres/Ollama genuinely need a writable root; the stateless services — `realtime`, `pii`, `copilot` — could tolerate it but aren't pre-wired with a `/tmp` `emptyDir`). If your policy requires it, set `.securityContext.readOnlyRootFilesystem: true` and mount an `emptyDir` at `/tmp` yourself via `extraVolumes`/`extraVolumeMounts`. * **Env validation** — keys under `app.env`, `realtime.env`, and `copilot.env` are passed through to the application and validated at startup. The JSON Schema intentionally does not enforce `additionalProperties: false` (would break custom user envs), so typos like `OPENA_API_KEY` (instead of `OPENAI_API_KEY`) surface as missing-key errors at runtime, not at `helm install` time. Review your env block carefully. * **Set public URLs** — `app.env.NEXT_PUBLIC_APP_URL` and `app.env.BETTER_AUTH_URL` must match your public origin (e.g. `https://sim.example.com`). Leaving them as `localhost` breaks sign-in. @@ -287,6 +295,17 @@ externalSecrets: INTERNAL_API_SECRET: sim/app/internal-api-secret postgresql: password: sim/postgresql/password + # Only needed when copilot.enabled=true and copilot.server.secret.create=true. + # Every non-empty copilot.server.env key must have a matching entry here — + # template rendering fails with a clear message naming the missing key otherwise. + copilot: + AGENT_API_DB_ENCRYPTION_KEY: sim/copilot/agent-api-db-encryption-key + INTERNAL_API_SECRET: sim/copilot/internal-api-secret + LICENSE_KEY: sim/copilot/license-key + SIM_BASE_URL: sim/copilot/sim-base-url + SIM_AGENT_API_KEY: sim/copilot/sim-agent-api-key + REDIS_URL: sim/copilot/redis-url + OPENAI_API_KEY_1: sim/copilot/openai-api-key ``` See `examples/values-external-secrets.yaml`. diff --git a/helm/sim/examples/values-external-secrets.yaml b/helm/sim/examples/values-external-secrets.yaml index 1ba38e1b960..23aa6c5cfbc 100644 --- a/helm/sim/examples/values-external-secrets.yaml +++ b/helm/sim/examples/values-external-secrets.yaml @@ -36,6 +36,16 @@ externalSecrets: API_ENCRYPTION_KEY: "sim/app/api-encryption-key" postgresql: password: "sim/postgresql/password" + # Only needed when copilot.enabled=true and copilot.server.secret.create=true + # (uncomment and pre-populate the referenced paths in your secret store): + # copilot: + # AGENT_API_DB_ENCRYPTION_KEY: "sim/copilot/agent-api-db-encryption-key" + # INTERNAL_API_SECRET: "sim/copilot/internal-api-secret" + # LICENSE_KEY: "sim/copilot/license-key" + # SIM_BASE_URL: "sim/copilot/sim-base-url" + # SIM_AGENT_API_KEY: "sim/copilot/sim-agent-api-key" + # REDIS_URL: "sim/copilot/redis-url" + # OPENAI_API_KEY_1: "sim/copilot/openai-api-key" app: enabled: true diff --git a/helm/sim/templates/_helpers.tpl b/helm/sim/templates/_helpers.tpl index 627b49b57a2..32d917a33b6 100644 --- a/helm/sim/templates/_helpers.tpl +++ b/helm/sim/templates/_helpers.tpl @@ -141,6 +141,38 @@ Migrations specific labels app.kubernetes.io/component: migrations {{- end }} +{{/* +Copilot specific labels +*/}} +{{- define "sim.copilot.labels" -}} +{{ include "sim.labels" . }} +app.kubernetes.io/component: copilot +{{- end }} + +{{/* +Copilot selector labels +*/}} +{{- define "sim.copilot.selectorLabels" -}} +{{ include "sim.selectorLabels" . }} +app.kubernetes.io/component: copilot +{{- end }} + +{{/* +Copilot PostgreSQL specific labels +*/}} +{{- define "sim.copilotPostgresql.labels" -}} +{{ include "sim.labels" . }} +app.kubernetes.io/component: copilot-postgresql +{{- end }} + +{{/* +Copilot PostgreSQL selector labels +*/}} +{{- define "sim.copilotPostgresql.selectorLabels" -}} +{{ include "sim.selectorLabels" . }} +app.kubernetes.io/component: copilot-postgresql +{{- end }} + {{/* Create the name of the service account to use */}} @@ -609,16 +641,33 @@ Validate Copilot configuration {{- end -}} {{- if .Values.copilot.server.secret.create -}} {{- $env := .Values.copilot.server.env -}} + {{- $useExternalSecrets := and .Values.externalSecrets .Values.externalSecrets.enabled -}} + {{- $remoteRefs := default (dict) (default (dict) .Values.externalSecrets.remoteRefs).copilot -}} {{- $required := list "AGENT_API_DB_ENCRYPTION_KEY" "INTERNAL_API_SECRET" "LICENSE_KEY" "SIM_BASE_URL" "SIM_AGENT_API_KEY" "REDIS_URL" -}} {{- range $key := $required -}} - {{- if not (and $env (index $env $key) (ne (index $env $key) "")) -}} - {{- fail (printf "copilot.server.env.%s is required when copilot is enabled" $key) -}} + {{- $inEnv := and $env (index $env $key) (ne (index $env $key) "") -}} + {{- $mapped := index $remoteRefs $key -}} + {{- if not (or $inEnv (and $useExternalSecrets $mapped)) -}} + {{- if $useExternalSecrets -}} + {{- fail (printf "Required key '%s' is missing: externalSecrets.enabled=true but the key is neither set in copilot.server.env nor mapped in externalSecrets.remoteRefs.copilot. Map it via externalSecrets.remoteRefs.copilot.%s='path/in/store' so it is synced into the copilot Secret." $key $key) -}} + {{- else -}} + {{- fail (printf "copilot.server.env.%s is required when copilot is enabled" $key) -}} + {{- end -}} + {{- end -}} + {{- end -}} + {{- if $useExternalSecrets -}} + {{- range $key, $value := $env -}} + {{- if and (ne (toString $value) "") (ne (toString $value) "") -}} + {{- if not (index $remoteRefs $key) -}} + {{- fail (printf "Key '%s' is set in copilot.server.env but externalSecrets.enabled=true and externalSecrets.remoteRefs.copilot.%s is not configured. When ESO is enabled the chart-managed copilot Secret is not rendered, so the container would start with no value. Either map it via externalSecrets.remoteRefs.copilot.%s='path/in/store' or remove it from copilot.server.env." $key $key $key) -}} + {{- end -}} + {{- end -}} {{- end -}} {{- end -}} - {{- $hasOpenAI := and $env (ne (default "" (index $env "OPENAI_API_KEY_1")) "") -}} - {{- $hasAnthropic := and $env (ne (default "" (index $env "ANTHROPIC_API_KEY_1")) "") -}} + {{- $hasOpenAI := or (and $env (ne (default "" (index $env "OPENAI_API_KEY_1")) "")) (and $useExternalSecrets (index $remoteRefs "OPENAI_API_KEY_1")) -}} + {{- $hasAnthropic := or (and $env (ne (default "" (index $env "ANTHROPIC_API_KEY_1")) "")) (and $useExternalSecrets (index $remoteRefs "ANTHROPIC_API_KEY_1")) -}} {{- if not (or $hasOpenAI $hasAnthropic) -}} - {{- fail "Set at least one of copilot.server.env.OPENAI_API_KEY_1 or copilot.server.env.ANTHROPIC_API_KEY_1" -}} + {{- fail "Set at least one of copilot.server.env.OPENAI_API_KEY_1 or copilot.server.env.ANTHROPIC_API_KEY_1 (or map one via externalSecrets.remoteRefs.copilot when externalSecrets.enabled=true)" -}} {{- end -}} {{- end -}} {{- if .Values.copilot.postgresql.enabled -}} diff --git a/helm/sim/templates/deployment-copilot.yaml b/helm/sim/templates/deployment-copilot.yaml index 6bab3f27ec2..900e140fc14 100644 --- a/helm/sim/templates/deployment-copilot.yaml +++ b/helm/sim/templates/deployment-copilot.yaml @@ -6,8 +6,7 @@ metadata: name: {{ include "sim.fullname" . }}-copilot namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.labels" . | nindent 4 }} spec: type: {{ .Values.copilot.server.service.type }} ports: @@ -16,9 +15,7 @@ spec: protocol: TCP name: http selector: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.selectorLabels" . | nindent 4 }} --- apiVersion: apps/v1 kind: Deployment @@ -26,15 +23,12 @@ metadata: name: {{ include "sim.fullname" . }}-copilot namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.labels" . | nindent 4 }} spec: replicas: {{ .Values.copilot.server.replicaCount }} selector: matchLabels: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.selectorLabels" . | nindent 6 }} template: metadata: annotations: @@ -43,9 +37,7 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} labels: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.selectorLabels" . | nindent 8 }} {{- with .Values.podLabels }} {{- toYaml . | nindent 8 }} {{- end }} @@ -106,5 +98,23 @@ spec: {{- toYaml . | nindent 12 }} {{- end }} {{- include "sim.containerSecurityContext" .Values.copilot.server | nindent 10 }} + {{- if or .Values.extraVolumeMounts .Values.copilot.server.extraVolumeMounts }} + volumeMounts: + {{- with .Values.extraVolumeMounts }} + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.copilot.server.extraVolumeMounts }} + {{- toYaml . | nindent 12 }} + {{- end }} + {{- end }} + {{- if or .Values.extraVolumes .Values.copilot.server.extraVolumes }} + volumes: + {{- with .Values.extraVolumes }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.copilot.server.extraVolumes }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} {{- end }} diff --git a/helm/sim/templates/external-secret-copilot.yaml b/helm/sim/templates/external-secret-copilot.yaml new file mode 100644 index 00000000000..a7b78da0310 --- /dev/null +++ b/helm/sim/templates/external-secret-copilot.yaml @@ -0,0 +1,38 @@ +{{- if and .Values.externalSecrets.enabled .Values.copilot.enabled .Values.copilot.server.secret.create }} +# ExternalSecret for copilot server credentials (syncs from external secret managers) +# +# Mirrors external-secret-app.yaml: the data list is generated by iterating every +# non-empty entry in externalSecrets.remoteRefs.copilot. sim.copilot.validate fails +# template rendering if a required key (or any non-empty copilot.server.env key) is +# missing a matching remoteRefs.copilot entry, so a misconfiguration surfaces at +# `helm template` time instead of a container starting with an empty value. +apiVersion: external-secrets.io/{{ .Values.externalSecrets.apiVersion | default "v1beta1" }} +kind: ExternalSecret +metadata: + name: {{ include "sim.copilot.envSecretName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "sim.copilot.labels" . | nindent 4 }} +spec: + refreshInterval: {{ .Values.externalSecrets.refreshInterval | quote }} + secretStoreRef: + name: {{ required "externalSecrets.secretStoreRef.name is required when externalSecrets.enabled=true" .Values.externalSecrets.secretStoreRef.name }} + kind: {{ .Values.externalSecrets.secretStoreRef.kind | default "ClusterSecretStore" }} + target: + name: {{ include "sim.copilot.envSecretName" . }} + creationPolicy: Owner + data: + {{- range $secretKey, $ref := .Values.externalSecrets.remoteRefs.copilot }} + {{- if $ref }} + {{- if kindIs "string" $ref }} + - secretKey: {{ $secretKey }} + remoteRef: + key: {{ $ref }} + {{- else if kindIs "map" $ref }} + - secretKey: {{ $secretKey }} + remoteRef: + {{- toYaml $ref | nindent 8 }} + {{- end }} + {{- end }} + {{- end }} +{{- end }} diff --git a/helm/sim/templates/poddisruptionbudget.yaml b/helm/sim/templates/poddisruptionbudget.yaml index 49c5015e786..7981fbda931 100644 --- a/helm/sim/templates/poddisruptionbudget.yaml +++ b/helm/sim/templates/poddisruptionbudget.yaml @@ -75,8 +75,7 @@ metadata: name: {{ include "sim.fullname" $ }}-copilot-pdb namespace: {{ $.Release.Namespace }} labels: - {{- include "sim.labels" $ | nindent 4 }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.labels" $ | nindent 4 }} spec: {{- if .minAvailable }} minAvailable: {{ .minAvailable }} @@ -90,8 +89,6 @@ spec: {{- end }} selector: matchLabels: - app.kubernetes.io/name: {{ include "sim.name" $ }} - app.kubernetes.io/instance: {{ $.Release.Name }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.selectorLabels" $ | nindent 6 }} {{- end }} {{- end }} diff --git a/helm/sim/templates/secrets-copilot.yaml b/helm/sim/templates/secrets-copilot.yaml index 55e397fdf3d..3dd9170055f 100644 --- a/helm/sim/templates/secrets-copilot.yaml +++ b/helm/sim/templates/secrets-copilot.yaml @@ -1,12 +1,11 @@ -{{- if and .Values.copilot.enabled .Values.copilot.server.secret.create }} +{{- if and .Values.copilot.enabled .Values.copilot.server.secret.create (not (and .Values.externalSecrets .Values.externalSecrets.enabled)) }} apiVersion: v1 kind: Secret metadata: name: {{ include "sim.copilot.envSecretName" . }} namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.labels" . | nindent 4 }} {{- with .Values.copilot.server.secret.annotations }} annotations: {{- toYaml . | nindent 4 }} @@ -25,8 +24,7 @@ metadata: name: {{ include "sim.copilot.databaseSecretName" . }} namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.labels" . | nindent 4 }} {{- with .Values.copilot.server.secret.annotations }} annotations: {{- toYaml . | nindent 4 }} diff --git a/helm/sim/templates/statefulset-copilot-postgres.yaml b/helm/sim/templates/statefulset-copilot-postgres.yaml index 91dd5bad195..fcb2e797774 100644 --- a/helm/sim/templates/statefulset-copilot-postgres.yaml +++ b/helm/sim/templates/statefulset-copilot-postgres.yaml @@ -5,8 +5,7 @@ metadata: name: {{ include "sim.fullname" . }}-copilot-postgresql-secret namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.labels" . | nindent 4 }} type: Opaque stringData: POSTGRES_USER: {{ .Values.copilot.postgresql.auth.username | quote }} @@ -21,8 +20,7 @@ metadata: name: {{ include "sim.fullname" . }}-copilot-postgresql namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.labels" . | nindent 4 }} spec: type: {{ .Values.copilot.postgresql.service.type }} ports: @@ -31,9 +29,7 @@ spec: protocol: TCP name: postgresql selector: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.selectorLabels" . | nindent 4 }} --- # Headless Service for the StatefulSet (stable per-pod DNS) apiVersion: v1 @@ -42,8 +38,7 @@ metadata: name: {{ include "sim.fullname" . }}-copilot-postgresql-headless namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.labels" . | nindent 4 }} spec: clusterIP: None publishNotReadyAddresses: true @@ -53,9 +48,7 @@ spec: protocol: TCP name: postgresql selector: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.selectorLabels" . | nindent 4 }} --- apiVersion: apps/v1 kind: StatefulSet @@ -63,8 +56,7 @@ metadata: name: {{ include "sim.fullname" . }}-copilot-postgresql namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.labels" . | nindent 4 }} spec: # Must remain {{ include "sim.fullname" . }}-copilot-postgresql (not the # -headless name) — spec.serviceName is immutable on a StatefulSet, and @@ -77,15 +69,11 @@ spec: type: RollingUpdate selector: matchLabels: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.selectorLabels" . | nindent 6 }} template: metadata: labels: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.selectorLabels" . | nindent 8 }} {{- with .Values.podLabels }} {{- toYaml . | nindent 8 }} {{- end }} @@ -143,8 +131,7 @@ spec: - metadata: name: data labels: - {{- include "sim.labels" . | nindent 10 }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.labels" . | nindent 10 }} spec: accessModes: {{- range .Values.copilot.postgresql.persistence.accessModes }} diff --git a/helm/sim/templates/telemetry.yaml b/helm/sim/templates/telemetry.yaml index aa78f8acd5c..71bfea320cf 100644 --- a/helm/sim/templates/telemetry.yaml +++ b/helm/sim/templates/telemetry.yaml @@ -125,10 +125,8 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} serviceAccountName: {{ include "sim.serviceAccountName" . }} - securityContext: - runAsNonRoot: true - runAsUser: 10001 - fsGroup: 10001 + automountServiceAccountToken: false + {{- include "sim.podSecurityContext" .Values.telemetry | nindent 6 }} containers: - name: otel-collector image: {{ include "sim.image" (dict "imageRoot" .Values.telemetry.image "global" .Values.global "chartAppVersion" .Chart.AppVersion) }} @@ -177,6 +175,7 @@ spec: failureThreshold: 3 resources: {{- toYaml .Values.telemetry.resources | nindent 12 }} + {{- include "sim.containerSecurityContext" .Values.telemetry | nindent 10 }} volumes: - name: otel-config configMap: diff --git a/helm/sim/tests/chart-computed-env_test.yaml b/helm/sim/tests/chart-computed-env_test.yaml index 50ac5206405..8d78fec65ab 100644 --- a/helm/sim/tests/chart-computed-env_test.yaml +++ b/helm/sim/tests/chart-computed-env_test.yaml @@ -7,12 +7,12 @@ tests: - it: app pod uses chart-computed DATABASE_URL even when user tries to override template: deployment-app.yaml set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x app.env.DATABASE_URL: "should-be-ignored" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - notContains: path: spec.template.spec.containers[0].env @@ -27,7 +27,7 @@ tests: app.secrets.existingSecret.name: my-secret app.env.DATABASE_URL: "postgres://evil-1:5432/x" app.env.SOCKET_SERVER_URL: "https://evil-2.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - notContains: path: spec.template.spec.containers[0].env @@ -43,7 +43,7 @@ tests: app.secrets.existingSecret.name: my-secret app.env.DATABASE_URL: "postgres://evil-1:5432/x" app.env.SOCKET_SERVER_URL: "https://evil-2.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - notContains: path: spec.template.spec.containers[0].env diff --git a/helm/sim/tests/copilot-secret-modes_test.yaml b/helm/sim/tests/copilot-secret-modes_test.yaml new file mode 100644 index 00000000000..81c3677910e --- /dev/null +++ b/helm/sim/tests/copilot-secret-modes_test.yaml @@ -0,0 +1,189 @@ +suite: copilot secret modes — inline / existingSecret / ESO routing (best-practices fix regression net) +release: + name: t + namespace: sim + +tests: + - it: inline mode renders the chart-managed copilot Secret + template: secrets-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: true + copilot.postgresql.auth.password: xxxxxxxx + copilot.server.env.AGENT_API_DB_ENCRYPTION_KEY: x + copilot.server.env.INTERNAL_API_SECRET: x + copilot.server.env.LICENSE_KEY: x + copilot.server.env.SIM_BASE_URL: x + copilot.server.env.SIM_AGENT_API_KEY: x + copilot.server.env.REDIS_URL: x + copilot.server.env.OPENAI_API_KEY_1: x + asserts: + - hasDocuments: { count: 1 } + - isKind: { of: Secret } + - equal: { path: metadata.name, value: t-sim-copilot-env } + + - it: existingSecret mode skips the chart-managed copilot Secret and the ExternalSecret + templates: + - secrets-copilot.yaml + - external-secret-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: true + copilot.postgresql.auth.password: xxxxxxxx + copilot.server.secret.create: false + copilot.server.secret.name: my-existing-copilot-secret + asserts: + - hasDocuments: { count: 0 } + + - it: ESO mode renders an ExternalSecret instead of the chart-managed copilot Secret + template: external-secret-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + externalSecrets.remoteRefs.copilot.PORT: path/to/port + externalSecrets.remoteRefs.copilot.SERVICE_NAME: path/to/name + externalSecrets.remoteRefs.copilot.ENVIRONMENT: path/to/env + externalSecrets.remoteRefs.copilot.LOG_LEVEL: path/to/log + externalSecrets.remoteRefs.copilot.AGENT_API_DB_ENCRYPTION_KEY: path/to/agentdbkey + externalSecrets.remoteRefs.copilot.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.copilot.LICENSE_KEY: path/to/license + externalSecrets.remoteRefs.copilot.SIM_BASE_URL: path/to/baseurl + externalSecrets.remoteRefs.copilot.SIM_AGENT_API_KEY: path/to/agentkey + externalSecrets.remoteRefs.copilot.REDIS_URL: path/to/redis + externalSecrets.remoteRefs.copilot.OPENAI_API_KEY_1: path/to/openai + asserts: + - isKind: { of: ExternalSecret } + - equal: { path: metadata.name, value: t-sim-copilot-env } + - equal: { path: spec.secretStoreRef.name, value: sim-store } + + - it: ESO mode skips the chart-managed copilot Secret + template: secrets-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: true + copilot.postgresql.auth.password: xxxxxxxx + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + externalSecrets.remoteRefs.copilot.PORT: path/to/port + externalSecrets.remoteRefs.copilot.SERVICE_NAME: path/to/name + externalSecrets.remoteRefs.copilot.ENVIRONMENT: path/to/env + externalSecrets.remoteRefs.copilot.LOG_LEVEL: path/to/log + externalSecrets.remoteRefs.copilot.AGENT_API_DB_ENCRYPTION_KEY: path/to/agentdbkey + externalSecrets.remoteRefs.copilot.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.copilot.LICENSE_KEY: path/to/license + externalSecrets.remoteRefs.copilot.SIM_BASE_URL: path/to/baseurl + externalSecrets.remoteRefs.copilot.SIM_AGENT_API_KEY: path/to/agentkey + externalSecrets.remoteRefs.copilot.REDIS_URL: path/to/redis + externalSecrets.remoteRefs.copilot.OPENAI_API_KEY_1: path/to/openai + asserts: + - hasDocuments: { count: 0 } + + - it: ESO validator fails when a required copilot key is neither in env nor mapped + template: deployment-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + asserts: + - failedTemplate: + errorPattern: "Required key 'AGENT_API_DB_ENCRYPTION_KEY' is missing" + + - it: ESO validator fails when copilot.server.env contains an unmapped key + template: deployment-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + externalSecrets.remoteRefs.copilot.PORT: path/to/port + externalSecrets.remoteRefs.copilot.SERVICE_NAME: path/to/name + externalSecrets.remoteRefs.copilot.ENVIRONMENT: path/to/env + externalSecrets.remoteRefs.copilot.LOG_LEVEL: path/to/log + externalSecrets.remoteRefs.copilot.AGENT_API_DB_ENCRYPTION_KEY: path/to/agentdbkey + externalSecrets.remoteRefs.copilot.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.copilot.LICENSE_KEY: path/to/license + externalSecrets.remoteRefs.copilot.SIM_BASE_URL: path/to/baseurl + externalSecrets.remoteRefs.copilot.SIM_AGENT_API_KEY: path/to/agentkey + externalSecrets.remoteRefs.copilot.REDIS_URL: path/to/redis + externalSecrets.remoteRefs.copilot.OPENAI_API_KEY_1: path/to/openai + copilot.server.env.UNMAPPED_KEY: "would-go-nowhere" + asserts: + - failedTemplate: + errorPattern: "Key 'UNMAPPED_KEY' is set in copilot.server.env but externalSecrets.enabled=true" + + - it: non-ESO mode still requires OPENAI_API_KEY_1 or ANTHROPIC_API_KEY_1 directly in env + template: deployment-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + copilot.server.env.AGENT_API_DB_ENCRYPTION_KEY: x + copilot.server.env.INTERNAL_API_SECRET: x + copilot.server.env.LICENSE_KEY: x + copilot.server.env.SIM_BASE_URL: x + copilot.server.env.SIM_AGENT_API_KEY: x + copilot.server.env.REDIS_URL: x + asserts: + - failedTemplate: + errorPattern: "Set at least one of copilot.server.env.OPENAI_API_KEY_1 or copilot.server.env.ANTHROPIC_API_KEY_1" diff --git a/helm/sim/tests/env-defaults_test.yaml b/helm/sim/tests/env-defaults_test.yaml index a658f3dbfb2..7e3fdcb4898 100644 --- a/helm/sim/tests/env-defaults_test.yaml +++ b/helm/sim/tests/env-defaults_test.yaml @@ -3,12 +3,11 @@ release: name: t namespace: sim defaults: &defaults - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x - + postgresql.auth.password: xxxxxxxx tests: - it: inline mode renders localhost envDefaults on the app pod template: deployment-app.yaml @@ -37,7 +36,7 @@ tests: set: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-secret - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - notContains: path: spec.template.spec.containers[0].env @@ -50,7 +49,7 @@ tests: set: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-secret - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - notContains: path: spec.template.spec.containers[0].env @@ -64,7 +63,7 @@ tests: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-secret app.env.NEXT_PUBLIC_APP_URL: "https://prod.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - contains: path: spec.template.spec.containers[0].env @@ -78,7 +77,7 @@ tests: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-secret app.env.NEXT_PUBLIC_APP_URL: "https://prod.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - contains: path: spec.template.spec.containers[0].env @@ -93,7 +92,7 @@ tests: app.secrets.existingSecret.name: my-secret app.env.NEXT_PUBLIC_APP_URL: "https://prod.example.com" realtime.env.NEXT_PUBLIC_APP_URL: "https://realtime.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - contains: path: spec.template.spec.containers[0].env diff --git a/helm/sim/tests/helm-test-hook_test.yaml b/helm/sim/tests/helm-test-hook_test.yaml index 4240d584ec2..ee19f4f7776 100644 --- a/helm/sim/tests/helm-test-hook_test.yaml +++ b/helm/sim/tests/helm-test-hook_test.yaml @@ -3,12 +3,11 @@ release: name: t namespace: sim defaults: &defaults - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x - + postgresql.auth.password: xxxxxxxx tests: - it: renders the test pod with helm.sh/hook=test by default template: tests/test-connection.yaml diff --git a/helm/sim/tests/networkpolicy_test.yaml b/helm/sim/tests/networkpolicy_test.yaml index d6e0b260ec4..355fab6d27d 100644 --- a/helm/sim/tests/networkpolicy_test.yaml +++ b/helm/sim/tests/networkpolicy_test.yaml @@ -5,11 +5,11 @@ release: name: t namespace: sim defaults: &defaults - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx networkPolicy.enabled: true ingress.enabled: true ingress.app.host: a.test diff --git a/helm/sim/tests/pdb-hpa_test.yaml b/helm/sim/tests/pdb-hpa_test.yaml index 70f3b0903a2..3960f2e4006 100644 --- a/helm/sim/tests/pdb-hpa_test.yaml +++ b/helm/sim/tests/pdb-hpa_test.yaml @@ -3,12 +3,11 @@ release: name: t namespace: sim defaults: &defaults - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x - + postgresql.auth.password: xxxxxxxx tests: - it: PDB does not render when replicaCount=1 and HPA is off (tri-state null) template: poddisruptionbudget.yaml diff --git a/helm/sim/tests/pii_test.yaml b/helm/sim/tests/pii_test.yaml index 3a416237207..f35c87a5759 100644 --- a/helm/sim/tests/pii_test.yaml +++ b/helm/sim/tests/pii_test.yaml @@ -3,12 +3,11 @@ release: name: t namespace: sim set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x - + postgresql.auth.password: xxxxxxxx tests: - it: does not render the pii deployment when disabled template: deployment-pii.yaml diff --git a/helm/sim/tests/pod-rollout_test.yaml b/helm/sim/tests/pod-rollout_test.yaml index 39b0c8c72fc..999ba85c434 100644 --- a/helm/sim/tests/pod-rollout_test.yaml +++ b/helm/sim/tests/pod-rollout_test.yaml @@ -3,12 +3,11 @@ release: name: t namespace: sim defaults: &defaults - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x - + postgresql.auth.password: xxxxxxxx tests: - it: app pod template has checksum/secret annotation template: deployment-app.yaml @@ -30,7 +29,7 @@ tests: template: deployment-app.yaml set: <<: *defaults - app.env.BETTER_AUTH_SECRET: original-value + app.env.BETTER_AUTH_SECRET: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy asserts: - notEqual: path: spec.template.metadata.annotations["checksum/secret"] diff --git a/helm/sim/tests/schema-secret-length_test.yaml b/helm/sim/tests/schema-secret-length_test.yaml new file mode 100644 index 00000000000..b2255d93853 --- /dev/null +++ b/helm/sim/tests/schema-secret-length_test.yaml @@ -0,0 +1,59 @@ +suite: values.schema.json — secret length enforcement (blocker fix regression net) +release: + name: t + namespace: sim + +tests: + - it: rejects a BETTER_AUTH_SECRET shorter than 32 characters + set: + app.env.BETTER_AUTH_SECRET: short + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + asserts: + - failedTemplate: + errorPattern: "BETTER_AUTH_SECRET.*minLength" + + - it: rejects an ENCRYPTION_KEY shorter than 32 characters + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: short + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + asserts: + - failedTemplate: + errorPattern: "ENCRYPTION_KEY.*minLength" + + - it: rejects a postgresql.auth.password shorter than 8 characters + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: short + asserts: + - failedTemplate: + errorPattern: "password.*minLength" + + - it: accepts an empty BETTER_AUTH_SECRET (deferred to existingSecret/ESO) + templates: + - secrets-app.yaml + set: + app.secrets.existingSecret.enabled: true + app.secrets.existingSecret.name: my-existing-secret + postgresql.auth.password: xxxxxxxx + asserts: + - hasDocuments: { count: 0 } + + - it: accepts a valid 32-character BETTER_AUTH_SECRET and ENCRYPTION_KEY + template: deployment-app.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + asserts: + - isKind: { of: Deployment } diff --git a/helm/sim/tests/secret-modes_test.yaml b/helm/sim/tests/secret-modes_test.yaml index d1e73c3fdab..6b60cf56693 100644 --- a/helm/sim/tests/secret-modes_test.yaml +++ b/helm/sim/tests/secret-modes_test.yaml @@ -7,11 +7,11 @@ tests: - it: inline mode renders the chart-managed Secret template: secrets-app.yaml set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - isKind: { of: Secret } - equal: { path: metadata.name, value: t-sim-app-secrets } @@ -23,7 +23,7 @@ tests: set: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-existing-secret - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - hasDocuments: { count: 0 } @@ -38,7 +38,7 @@ tests: externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron externalSecrets.remoteRefs.postgresql.password: path/to/pgpw - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - isKind: { of: ExternalSecret } - equal: { path: metadata.name, value: t-sim-app-secrets } @@ -55,7 +55,7 @@ tests: externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron externalSecrets.remoteRefs.postgresql.password: path/to/pgpw - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - hasDocuments: { count: 0 } @@ -69,7 +69,7 @@ tests: externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron externalSecrets.remoteRefs.postgresql.password: path/to/pgpw app.env.UNMAPPED_KEY: "would-go-nowhere" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorPattern: "Key 'UNMAPPED_KEY' is set in app.env but externalSecrets.enabled=true" @@ -79,7 +79,7 @@ tests: set: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-existing-secret - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - contains: path: spec.template.spec.containers[0].envFrom @@ -90,13 +90,13 @@ tests: - it: realtime.env-only value reaches the Secret when app.env entry is empty (cursor bugbot fix) template: secrets-app.yaml set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x app.env.BETTER_AUTH_URL: "" realtime.env.BETTER_AUTH_URL: "https://realtime.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - equal: path: stringData.BETTER_AUTH_URL @@ -105,13 +105,13 @@ tests: - it: app.env wins over realtime.env on collision when both are non-empty template: secrets-app.yaml set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x app.env.NEXT_PUBLIC_APP_URL: "https://app.example.com" realtime.env.NEXT_PUBLIC_APP_URL: "https://realtime.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - equal: path: stringData.NEXT_PUBLIC_APP_URL diff --git a/helm/sim/tests/smoke_test.yaml b/helm/sim/tests/smoke_test.yaml index 71bf0888236..0e50a2799be 100644 --- a/helm/sim/tests/smoke_test.yaml +++ b/helm/sim/tests/smoke_test.yaml @@ -9,7 +9,7 @@ release: namespace: sim set: app.env.BETTER_AUTH_SECRET: test-better-auth-secret-32-bytes - app.env.ENCRYPTION_KEY: test-encryption-key-32-bytes + app.env.ENCRYPTION_KEY: test-encryption-key-32-bytes-long app.env.INTERNAL_API_SECRET: test-internal-api-secret-32-bytes app.env.CRON_SECRET: test-cron-secret-32-bytes postgresql.auth.password: testpassword diff --git a/helm/sim/tests/telemetry-security-context_test.yaml b/helm/sim/tests/telemetry-security-context_test.yaml new file mode 100644 index 00000000000..4d3f7233899 --- /dev/null +++ b/helm/sim/tests/telemetry-security-context_test.yaml @@ -0,0 +1,53 @@ +suite: telemetry — Restricted Pod Security Standard hardening (best-practices fix regression net) +release: + name: t + namespace: sim +templates: + - telemetry.yaml + +tests: + - it: pod-level security context preserves the collector's original UID/GID/fsGroup and adds seccompProfile + set: + telemetry.enabled: true + documentSelector: + path: kind + value: Deployment + asserts: + - equal: + path: spec.template.spec.securityContext.runAsUser + value: 10001 + - equal: + path: spec.template.spec.securityContext.runAsGroup + value: 10001 + - equal: + path: spec.template.spec.securityContext.fsGroup + value: 10001 + - equal: + path: spec.template.spec.securityContext.runAsNonRoot + value: true + - equal: + path: spec.template.spec.securityContext.seccompProfile.type + value: RuntimeDefault + - equal: + path: spec.template.spec.automountServiceAccountToken + value: false + + - it: container-level security context matches the Restricted profile (same as every other workload) + set: + telemetry.enabled: true + documentSelector: + path: kind + value: Deployment + asserts: + - equal: + path: spec.template.spec.containers[0].securityContext.allowPrivilegeEscalation + value: false + - equal: + path: spec.template.spec.containers[0].securityContext.capabilities.drop[0] + value: ALL + - equal: + path: spec.template.spec.containers[0].securityContext.runAsNonRoot + value: true + - equal: + path: spec.template.spec.containers[0].securityContext.seccompProfile.type + value: RuntimeDefault diff --git a/helm/sim/tests/validators_test.yaml b/helm/sim/tests/validators_test.yaml index b53800b2d14..aff09b4386b 100644 --- a/helm/sim/tests/validators_test.yaml +++ b/helm/sim/tests/validators_test.yaml @@ -9,52 +9,52 @@ tests: - it: fails when BETTER_AUTH_SECRET is missing set: app.env.BETTER_AUTH_SECRET: "" - app.env.ENCRYPTION_KEY: x + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorMessage: "app.env.BETTER_AUTH_SECRET is required for production deployment" - it: fails when ENCRYPTION_KEY is missing set: - app.env.BETTER_AUTH_SECRET: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.ENCRYPTION_KEY: "" app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorMessage: "app.env.ENCRYPTION_KEY is required for production deployment" - it: fails when INTERNAL_API_SECRET is missing (1.0.0 requirement) set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: "" app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorMessage: "app.env.INTERNAL_API_SECRET is required for production deployment (shared auth between sim-app and sim-realtime pods). Generate one with: openssl rand -hex 32" - it: fails when cronjobs are enabled but CRON_SECRET is empty set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: "" cronjobs.enabled: true - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorMessage: "app.env.CRON_SECRET is required when cronjobs.enabled=true (every cron pod authenticates with this token). Generate one with: openssl rand -hex 32, or set cronjobs.enabled=false." - it: fails when postgresql.auth.password is missing set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x postgresql.auth.password: "" @@ -64,8 +64,8 @@ tests: - it: rejects postgres passwords with URL-unsafe characters set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x postgresql.auth.password: "pa$$word" @@ -76,10 +76,10 @@ tests: - it: rejects placeholder BETTER_AUTH_SECRET set: app.env.BETTER_AUTH_SECRET: "CHANGE-ME-32-CHAR-SECRET-FOR-PRODUCTION-USE" - app.env.ENCRYPTION_KEY: x + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorPattern: "must not use the default placeholder value" diff --git a/helm/sim/values.schema.json b/helm/sim/values.schema.json index f6641894638..4b1e86c0a75 100644 --- a/helm/sim/values.schema.json +++ b/helm/sim/values.schema.json @@ -108,11 +108,13 @@ "properties": { "BETTER_AUTH_SECRET": { "type": "string", - "description": "Auth secret (minimum 32 characters required when not using existingSecret)" + "anyOf": [{ "minLength": 32 }, { "const": "" }], + "description": "Auth secret (minimum 32 characters required when not using existingSecret; leave empty when the value comes from existingSecret/ESO instead)" }, "ENCRYPTION_KEY": { "type": "string", - "description": "Encryption key (minimum 32 characters required when not using existingSecret)" + "anyOf": [{ "minLength": 32 }, { "const": "" }], + "description": "Encryption key (minimum 32 characters required when not using existingSecret; leave empty when the value comes from existingSecret/ESO instead)" }, "NEXT_PUBLIC_APP_URL": { "type": "string", @@ -432,7 +434,8 @@ "properties": { "BETTER_AUTH_SECRET": { "type": "string", - "description": "Auth secret (minimum 32 characters required when not using existingSecret)" + "anyOf": [{ "minLength": 32 }, { "const": "" }], + "description": "Auth secret (minimum 32 characters required when not using existingSecret; leave empty when the value comes from existingSecret/ESO instead)" }, "NEXT_PUBLIC_APP_URL": { "type": "string", @@ -528,7 +531,8 @@ }, "password": { "type": "string", - "description": "PostgreSQL password (minimum 8 characters when not using existingSecret)" + "anyOf": [{ "minLength": 8 }, { "const": "" }], + "description": "PostgreSQL password (minimum 8 characters when not using existingSecret; leave empty when the value comes from existingSecret/ESO instead)" }, "existingSecret": { "type": "object", diff --git a/helm/sim/values.yaml b/helm/sim/values.yaml index 592d2d4f4f7..1f9b2acc796 100644 --- a/helm/sim/values.yaml +++ b/helm/sim/values.yaml @@ -1437,6 +1437,15 @@ telemetry: tls: enabled: false + # Pod security context (preserves the collector's original UID/GID/fsGroup) + podSecurityContext: + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + + # Container security context (sim.containerSecurityContext restricted defaults apply) + securityContext: {} + # Copilot service configuration (optional microservice) copilot: # Enable/disable the copilot service @@ -1498,10 +1507,15 @@ copilot: # Optional: additional static environment variables extraEnv: [] - + # Optional: references to existing ConfigMaps/Secrets extraEnvFrom: [] - + + # Optional: additional volumes/mounts (e.g. to back an emptyDir /tmp when + # enabling securityContext.readOnlyRootFilesystem) + extraVolumes: [] + extraVolumeMounts: [] + # Secret generation configuration (set create=false to use an existing secret) secret: create: true @@ -1727,6 +1741,27 @@ externalSecrets: # Path to external database password in external store password: "" + # Copilot server secrets (used when copilot.enabled=true and copilot.server.secret.create=true). + # Every non-empty copilot.server.env key must have a matching entry here, or template + # rendering fails with a clear "map it or remove it" error — same rule as remoteRefs.app. + copilot: + # Path to AGENT_API_DB_ENCRYPTION_KEY in external store + AGENT_API_DB_ENCRYPTION_KEY: "" + # Path to INTERNAL_API_SECRET in external store (shared with app/realtime) + INTERNAL_API_SECRET: "" + # Path to LICENSE_KEY in external store + LICENSE_KEY: "" + # Path to SIM_BASE_URL in external store + SIM_BASE_URL: "" + # Path to SIM_AGENT_API_KEY in external store + SIM_AGENT_API_KEY: "" + # Path to REDIS_URL in external store + REDIS_URL: "" + # Path to OPENAI_API_KEY_1 in external store (one of OPENAI_API_KEY_1/ANTHROPIC_API_KEY_1 required) + OPENAI_API_KEY_1: "" + # Path to ANTHROPIC_API_KEY_1 in external store (one of OPENAI_API_KEY_1/ANTHROPIC_API_KEY_1 required) + ANTHROPIC_API_KEY_1: "" + # cert-manager configuration # Prerequisites: Install cert-manager in your cluster first # See: https://cert-manager.io/docs/installation/ From a28efb9a6f006fc767e7e15e47f05e8134ae34ea Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Thu, 9 Jul 2026 19:06:00 -0700 Subject: [PATCH 2/5] fix(helm): move copilot's static config out of the ESO-required Secret MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Greptile caught a real bug: copilot.server.env shipped with non-empty static defaults (PORT, SERVICE_NAME, ENVIRONMENT, LOG_LEVEL), unlike app.env/realtime.env which ship fully empty. The new ESO validation correctly required every non-empty env key to be mapped in externalSecrets.remoteRefs.copilot — but that meant a default install with copilot + ESO enabled failed demanding secret-store paths for values that were never secrets. Fixed by applying the chart's own existing pattern for this exact problem: moved the 4 static keys into copilot.server.envDefaults (mirroring app.envDefaults) and inlined them as plain container env, bypassing the Secret/ExternalSecret system entirely — same rationale already documented for app.envDefaults. Verified live that Greptile's exact repro (default copilot env + ESO enabled, only the 7 real secrets mapped) now renders cleanly and the four values still reach the container. Added a regression test that fails on the pre-fix code. --- helm/sim/examples/values-copilot.yaml | 12 +++-- helm/sim/templates/deployment-copilot.yaml | 18 +++++++- helm/sim/tests/copilot-secret-modes_test.yaml | 44 +++++++++++++++++++ helm/sim/values.yaml | 24 +++++++--- 4 files changed, 85 insertions(+), 13 deletions(-) diff --git a/helm/sim/examples/values-copilot.yaml b/helm/sim/examples/values-copilot.yaml index 62fbb87b1a6..d95e1aa2e91 100644 --- a/helm/sim/examples/values-copilot.yaml +++ b/helm/sim/examples/values-copilot.yaml @@ -60,9 +60,6 @@ copilot: # Required secrets (set via values or provide your own secret) env: - PORT: "8080" - SERVICE_NAME: "copilot" - ENVIRONMENT: "production" AGENT_API_DB_ENCRYPTION_KEY: "" # openssl rand -hex 32 INTERNAL_API_SECRET: "" # reuse Sim INTERNAL_API_SECRET LICENSE_KEY: "" # Provided by Sim team @@ -72,9 +69,16 @@ copilot: SIM_AGENT_API_KEY: "" # Must match SIM-side COPILOT_API_KEY REDIS_URL: "redis://default:password@redis:6379" # Optional configuration - LOG_LEVEL: "info" CORS_ALLOWED_ORIGINS: "https://sim.example.com" OTEL_EXPORTER_OTLP_ENDPOINT: "" + + # Non-secret operational tunables — inlined as plain container env, never go through + # the Secret/ExternalSecret, so they don't need externalSecrets.remoteRefs.copilot entries + envDefaults: + PORT: "8080" + SERVICE_NAME: "copilot" + ENVIRONMENT: "production" + LOG_LEVEL: "info" # Create a Secret from the values above. Set create=false to reference an existing secret instead. secret: diff --git a/helm/sim/templates/deployment-copilot.yaml b/helm/sim/templates/deployment-copilot.yaml index 900e140fc14..e93ff137d02 100644 --- a/helm/sim/templates/deployment-copilot.yaml +++ b/helm/sim/templates/deployment-copilot.yaml @@ -77,10 +77,24 @@ spec: {{- with .Values.copilot.server.extraEnvFrom }} {{- toYaml . | nindent 12 }} {{- end }} - {{- with .Values.copilot.server.extraEnv }} + {{- /* + copilot.server.env flows through envFrom (the Secret/ExternalSecret) above. + copilot.server.envDefaults are inlined here so they don't leak into the + chart-managed Secret and don't need to be mapped when + externalSecrets.enabled=true — same rationale as app.envDefaults. + */}} env: + {{- $copilotEnv := .Values.copilot.server.env | default dict }} + {{- range $key, $value := .Values.copilot.server.envDefaults | default dict }} + {{- $override := index $copilotEnv $key }} + {{- if and (ne (toString $value) "") (ne (toString $value) "") (or (not $override) (eq (toString $override) "") (eq (toString $override) "")) }} + - name: {{ $key }} + value: {{ $value | quote }} + {{- end }} + {{- end }} + {{- with .Values.copilot.server.extraEnv }} {{- toYaml . | nindent 12 }} - {{- end }} + {{- end }} {{- if .Values.copilot.server.startupProbe }} startupProbe: {{- toYaml .Values.copilot.server.startupProbe | nindent 12 }} diff --git a/helm/sim/tests/copilot-secret-modes_test.yaml b/helm/sim/tests/copilot-secret-modes_test.yaml index 81c3677910e..4dd107d8dca 100644 --- a/helm/sim/tests/copilot-secret-modes_test.yaml +++ b/helm/sim/tests/copilot-secret-modes_test.yaml @@ -4,6 +4,50 @@ release: namespace: sim tests: + - it: ESO mode with only the required copilot secrets mapped renders cleanly (envDefaults regression net — Greptile round 1 finding) + template: deployment-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + # Only the genuinely-secret keys are mapped — PORT/SERVICE_NAME/ENVIRONMENT/LOG_LEVEL + # live in copilot.server.envDefaults now, not copilot.server.env, so they must NOT + # need a remoteRefs.copilot entry. + externalSecrets.remoteRefs.copilot.AGENT_API_DB_ENCRYPTION_KEY: path/to/agentdbkey + externalSecrets.remoteRefs.copilot.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.copilot.LICENSE_KEY: path/to/license + externalSecrets.remoteRefs.copilot.SIM_BASE_URL: path/to/baseurl + externalSecrets.remoteRefs.copilot.SIM_AGENT_API_KEY: path/to/agentkey + externalSecrets.remoteRefs.copilot.REDIS_URL: path/to/redis + externalSecrets.remoteRefs.copilot.OPENAI_API_KEY_1: path/to/openai + documentSelector: + path: kind + value: Deployment + asserts: + - isKind: { of: Deployment } + - contains: + path: spec.template.spec.containers[0].env + content: + name: PORT + value: "8080" + - contains: + path: spec.template.spec.containers[0].env + content: + name: SERVICE_NAME + value: copilot + - it: inline mode renders the chart-managed copilot Secret template: secrets-copilot.yaml set: diff --git a/helm/sim/values.yaml b/helm/sim/values.yaml index 1f9b2acc796..c7a39c81e92 100644 --- a/helm/sim/values.yaml +++ b/helm/sim/values.yaml @@ -1487,11 +1487,12 @@ copilot: runAsNonRoot: true runAsUser: 1001 - # Environment variables (required and optional) + # Environment variables that go through the chart-managed Secret (or ExternalSecret + # when externalSecrets.enabled=true). Every non-empty key here must have a matching + # externalSecrets.remoteRefs.copilot entry under ESO — reserve this block for values + # that are genuinely sensitive or user-provided; static operational config belongs in + # envDefaults below instead. env: - PORT: "8080" - SERVICE_NAME: "copilot" - ENVIRONMENT: "production" AGENT_API_DB_ENCRYPTION_KEY: "" INTERNAL_API_SECRET: "" LICENSE_KEY: "" @@ -1500,11 +1501,20 @@ copilot: SIM_BASE_URL: "" SIM_AGENT_API_KEY: "" REDIS_URL: "" - # Optional configuration - LOG_LEVEL: "info" CORS_ALLOWED_ORIGINS: "" OTEL_EXPORTER_OTLP_ENDPOINT: "" - + + # Operational tunables shipped with the chart. Rendered as inline `env:` on the + # copilot container — NOT written into the chart-managed Secret and NOT required to + # be mapped when externalSecrets.enabled=true. Override any key by setting + # copilot.server.envDefaults.KEY in your values file. Move a key into + # copilot.server.env above only if it must be treated as secret. + envDefaults: + PORT: "8080" + SERVICE_NAME: "copilot" + ENVIRONMENT: "production" + LOG_LEVEL: "info" + # Optional: additional static environment variables extraEnv: [] From 27d77433ad9d630c189168d1a43feba759318c4f Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Thu, 9 Jul 2026 19:13:09 -0700 Subject: [PATCH 3/5] fix(helm): don't shadow copilot's existingSecret with envDefaults MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Greptile and Cursor Bugbot both independently caught this: in copilot.server.secret.create=false (existingSecret) mode, the chart still unconditionally inlined copilot.server.envDefaults as explicit container env. Kubernetes gives explicit env precedence over envFrom, so a pre-existing Secret's PORT/LOG_LEVEL/etc values were silently overridden by the chart defaults — the exact shadowing bug app.envDefaults already guards against via its own $useExistingSecret skip, which I forgot to mirror when copying the pattern to copilot. Skip envDefaults entirely in existingSecret mode (matching app's existing behavior — the pre-created Secret is the sole source of truth), while still rendering extraEnv. Verified live: existingSecret mode now renders no env: block at all when extraEnv is unset, and still renders extraEnv without envDefaults leaking in when it is set. Added two regression tests, confirmed both fail on the pre-fix code. --- helm/sim/templates/deployment-copilot.yaml | 11 +++- helm/sim/tests/copilot-secret-modes_test.yaml | 51 +++++++++++++++++++ 2 files changed, 61 insertions(+), 1 deletion(-) diff --git a/helm/sim/templates/deployment-copilot.yaml b/helm/sim/templates/deployment-copilot.yaml index e93ff137d02..ec86fefc5a8 100644 --- a/helm/sim/templates/deployment-copilot.yaml +++ b/helm/sim/templates/deployment-copilot.yaml @@ -81,10 +81,17 @@ spec: copilot.server.env flows through envFrom (the Secret/ExternalSecret) above. copilot.server.envDefaults are inlined here so they don't leak into the chart-managed Secret and don't need to be mapped when - externalSecrets.enabled=true — same rationale as app.envDefaults. + externalSecrets.enabled=true — same rationale as app.envDefaults. Skipped + entirely in existingSecret mode: the pre-created Secret is the source of + truth, and inlining defaults would silently shadow keys (PORT, LOG_LEVEL, + etc.) the user stored there via envFrom, since explicit env wins over + envFrom. */}} + {{- $useExistingSecret := and (not .Values.externalSecrets.enabled) (not .Values.copilot.server.secret.create) }} + {{- if or (not $useExistingSecret) .Values.copilot.server.extraEnv }} env: {{- $copilotEnv := .Values.copilot.server.env | default dict }} + {{- if not $useExistingSecret }} {{- range $key, $value := .Values.copilot.server.envDefaults | default dict }} {{- $override := index $copilotEnv $key }} {{- if and (ne (toString $value) "") (ne (toString $value) "") (or (not $override) (eq (toString $override) "") (eq (toString $override) "")) }} @@ -92,9 +99,11 @@ spec: value: {{ $value | quote }} {{- end }} {{- end }} + {{- end }} {{- with .Values.copilot.server.extraEnv }} {{- toYaml . | nindent 12 }} {{- end }} + {{- end }} {{- if .Values.copilot.server.startupProbe }} startupProbe: {{- toYaml .Values.copilot.server.startupProbe | nindent 12 }} diff --git a/helm/sim/tests/copilot-secret-modes_test.yaml b/helm/sim/tests/copilot-secret-modes_test.yaml index 4dd107d8dca..eab026a8a8d 100644 --- a/helm/sim/tests/copilot-secret-modes_test.yaml +++ b/helm/sim/tests/copilot-secret-modes_test.yaml @@ -71,6 +71,57 @@ tests: - isKind: { of: Secret } - equal: { path: metadata.name, value: t-sim-copilot-env } + - it: existingSecret mode does not shadow the pre-created Secret's values with envDefaults (Greptile + Cursor round 2 finding) + template: deployment-copilot.yaml + documentSelector: + path: kind + value: Deployment + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + copilot.server.secret.create: false + copilot.server.secret.name: my-existing-copilot-secret + asserts: + - notExists: + path: spec.template.spec.containers[0].env + + - it: existingSecret mode still renders extraEnv even though envDefaults are skipped + template: deployment-copilot.yaml + documentSelector: + path: kind + value: Deployment + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + copilot.server.secret.create: false + copilot.server.secret.name: my-existing-copilot-secret + copilot.server.extraEnv: + - name: CUSTOM_VAR + value: custom-value + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: CUSTOM_VAR + value: custom-value + - notContains: + path: spec.template.spec.containers[0].env + content: + name: PORT + value: "8080" + - it: existingSecret mode skips the chart-managed copilot Secret and the ExternalSecret templates: - secrets-copilot.yaml From 80e931d8048aeb5aed99627da839087290744bff Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Thu, 9 Jul 2026 19:19:51 -0700 Subject: [PATCH 4/5] fix(helm): key copilot's existingSecret check off its own secret.create, not the global ESO flag MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 2's fix (which I copied nearly verbatim from Greptile's own suggested diff) used $useExistingSecret := and (not externalSecrets.enabled) (not copilot.server.secret.create) — Greptile caught its own suggestion's remaining bug on round 3: when externalSecrets.enabled=true globally (for app/postgresql) but copilot itself uses copilot.server.secret.create=false with its own pre-created Secret, that condition evaluated to non-existingSecret mode, so envDefaults still inlined and shadowed the user's Secret values — same bug, different trigger condition. envFrom always points at the user-provided Secret name whenever secret.create=false, independent of what other components do with ESO, so the check should key on that alone. Verified live: global ESO enabled + copilot's own existingSecret now renders no env: block and envFrom correctly points at the pre-created secret name; the two scenarios that should still inline (copilot itself on ESO, plain inline mode) still work. Added a regression test, confirmed it fails against round 2's guard. --- helm/sim/templates/deployment-copilot.yaml | 7 +++-- helm/sim/tests/copilot-secret-modes_test.yaml | 30 +++++++++++++++++++ 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/helm/sim/templates/deployment-copilot.yaml b/helm/sim/templates/deployment-copilot.yaml index ec86fefc5a8..dc6cceabcd0 100644 --- a/helm/sim/templates/deployment-copilot.yaml +++ b/helm/sim/templates/deployment-copilot.yaml @@ -85,9 +85,12 @@ spec: entirely in existingSecret mode: the pre-created Secret is the source of truth, and inlining defaults would silently shadow keys (PORT, LOG_LEVEL, etc.) the user stored there via envFrom, since explicit env wins over - envFrom. + envFrom. Keyed purely on copilot.server.secret.create, NOT the global + externalSecrets.enabled flag — envFrom always points at the user-provided + Secret name whenever secret.create=false, regardless of whether ESO is used + for other components, so the global flag is irrelevant here. */}} - {{- $useExistingSecret := and (not .Values.externalSecrets.enabled) (not .Values.copilot.server.secret.create) }} + {{- $useExistingSecret := not .Values.copilot.server.secret.create }} {{- if or (not $useExistingSecret) .Values.copilot.server.extraEnv }} env: {{- $copilotEnv := .Values.copilot.server.env | default dict }} diff --git a/helm/sim/tests/copilot-secret-modes_test.yaml b/helm/sim/tests/copilot-secret-modes_test.yaml index eab026a8a8d..35907de6b7b 100644 --- a/helm/sim/tests/copilot-secret-modes_test.yaml +++ b/helm/sim/tests/copilot-secret-modes_test.yaml @@ -122,6 +122,36 @@ tests: name: PORT value: "8080" + - it: copilot's own existingSecret is still the source of truth even when externalSecrets.enabled=true globally for other components (Greptile round 3 finding) + template: deployment-copilot.yaml + documentSelector: + path: kind + value: Deployment + set: + app.env.BETTER_AUTH_SECRET: "" + app.env.ENCRYPTION_KEY: "" + app.env.INTERNAL_API_SECRET: "" + app.env.CRON_SECRET: "" + postgresql.auth.password: "" + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + copilot.server.secret.create: false + copilot.server.secret.name: my-existing-copilot-secret + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + asserts: + - notExists: + path: spec.template.spec.containers[0].env + - equal: + path: spec.template.spec.containers[0].envFrom[0].secretRef.name + value: my-existing-copilot-secret + - it: existingSecret mode skips the chart-managed copilot Secret and the ExternalSecret templates: - secrets-copilot.yaml From fd364c27ff7921c9aa96a477083f738908643f0b Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Thu, 9 Jul 2026 19:27:30 -0700 Subject: [PATCH 5/5] fix(helm): checksum/secret annotation on copilot ignores ESO-sourced secret MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cursor Bugbot caught a real bug: checksum/secret only hashed secrets-copilot.yaml's rendered output, but under externalSecrets.enabled=true that template renders nothing (env credentials come from external-secret-copilot.yaml instead). Result: changing externalSecrets.remoteRefs.copilot mappings wouldn't change the pod template hash, so Kubernetes would never restart the copilot pod to pick up the new mapping — stale envFrom values until a manual restart. Fixed by hashing the concatenation of both templates' rendered output: whichever mode is active, only one renders non-empty content, but the concatenated hash still changes on a mode switch or a remoteRefs change. This can't reach into the live secret store value ESO syncs (Helm only sees the ExternalSecret manifest at render time) — that's an inherent ESO limitation, not something a checksum annotation can close; documented as such in the template comment. Note: deployment-app.yaml and deployment-realtime.yaml have this same latent limitation in ESO mode (checksum/secret only hashes secrets-app.yaml), but that's pre-existing code outside this PR's diff — not fixed here to stay scoped to what Cursor actually flagged. Verified live: the checksum differs across two different remoteRefs.copilot.LICENSE_KEY mappings, and still changes correctly in plain inline mode. Added a regression test. --- helm/sim/templates/deployment-copilot.yaml | 11 +++++- helm/sim/tests/copilot-secret-modes_test.yaml | 35 +++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/helm/sim/templates/deployment-copilot.yaml b/helm/sim/templates/deployment-copilot.yaml index dc6cceabcd0..985a323433e 100644 --- a/helm/sim/templates/deployment-copilot.yaml +++ b/helm/sim/templates/deployment-copilot.yaml @@ -32,7 +32,16 @@ spec: template: metadata: annotations: - checksum/secret: {{ include (print $.Template.BasePath "/secrets-copilot.yaml") . | sha256sum }} + {{- /* + Hash both the inline Secret and the ExternalSecret template — whichever mode is + active, only one renders non-empty content, but concatenating both means a mode + switch or a remoteRefs.copilot mapping change still changes the checksum and + forces a rollout. This can't see the live value ESO syncs from the external + store (Helm only has the ExternalSecret manifest at render time, not the + store's payload) — that's an inherent ESO limitation, not something a Helm + checksum can close; ESO's own reconciliation handles picking up rotated values. + */}} + checksum/secret: {{ printf "%s%s" (include (print $.Template.BasePath "/secrets-copilot.yaml") .) (include (print $.Template.BasePath "/external-secret-copilot.yaml") .) | sha256sum }} {{- with .Values.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} diff --git a/helm/sim/tests/copilot-secret-modes_test.yaml b/helm/sim/tests/copilot-secret-modes_test.yaml index 35907de6b7b..dd881a27195 100644 --- a/helm/sim/tests/copilot-secret-modes_test.yaml +++ b/helm/sim/tests/copilot-secret-modes_test.yaml @@ -4,6 +4,41 @@ release: namespace: sim tests: + - it: checksum/secret changes when an ESO remoteRefs.copilot mapping changes, forcing a rollout (Cursor round 4 finding) + template: deployment-copilot.yaml + documentSelector: + path: kind + value: Deployment + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + externalSecrets.remoteRefs.copilot.AGENT_API_DB_ENCRYPTION_KEY: path/to/agentdbkey + externalSecrets.remoteRefs.copilot.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.copilot.LICENSE_KEY: path/to/license-v1 + externalSecrets.remoteRefs.copilot.SIM_BASE_URL: path/to/baseurl + externalSecrets.remoteRefs.copilot.SIM_AGENT_API_KEY: path/to/agentkey + externalSecrets.remoteRefs.copilot.REDIS_URL: path/to/redis + externalSecrets.remoteRefs.copilot.OPENAI_API_KEY_1: path/to/openai + asserts: + - isNotEmpty: + path: spec.template.metadata.annotations["checksum/secret"] + - matchRegex: + path: spec.template.metadata.annotations["checksum/secret"] + pattern: "^[a-f0-9]{64}$" + - it: ESO mode with only the required copilot secrets mapped renders cleanly (envDefaults regression net — Greptile round 1 finding) template: deployment-copilot.yaml set: