Skip to content

fix(helm): close blocker/real-gap findings from Helm chart best-practices audit#5555

Merged
waleedlatif1 merged 5 commits into
stagingfrom
helm-best-practices-hardening
Jul 10, 2026
Merged

fix(helm): close blocker/real-gap findings from Helm chart best-practices audit#5555
waleedlatif1 merged 5 commits into
stagingfrom
helm-best-practices-hardening

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • Ran a full best-practices audit of the Helm chart (independent of the earlier Azure storage PR), verified every finding against the official Helm docs and Kubernetes Pod Security Standards docs, then fixed each real one
  • Blocker: values.schema.json claimed "minimum 32/8 characters required" on BETTER_AUTH_SECRET/ENCRYPTION_KEY/postgresql.auth.password but never enforced it — a 5-character secret passed validation. Added anyOf minLength-or-empty constraints (empty stays legal for existingSecret/ESO modes)
  • Real gap: copilot didn't support External Secrets Operator, unlike app/postgresql/externalDatabase. Added external-secret-copilot.yaml, remoteRefs.copilot, and the same "map it or remove it" fail-fast validation the rest of the chart has
  • Real gap: the OpenTelemetry Collector was the only workload missing container-level Restricted Pod Security Standard hardening (no allowPrivilegeEscalation: false, no capability drop, no seccomp). Wired in the shared security-context helpers, preserving its original UID/GID/fsGroup
  • Real gap: copilot templates hand-rolled label/selector blocks instead of the chart's established sim.<component>.labels pattern. Added the missing helpers and refactored every consumer — verified byte-identical helm template output before/after (selector labels are immutable on upgrade, so I checked this, not assumed it)
  • Documented the networkPolicy.ingressFrom default and readOnlyRootFilesystem posture in the README (both real, intentional tradeoffs the audit flagged as underdocumented), and added extraVolumes/extraVolumeMounts to copilot's Deployment so that guidance is actually actionable (realtime/pii already had it, copilot didn't)
  • Deferred as non-blocking: pinning two floating Postgres image tags, schema stubs for ~13 uncovered sections, an OTel collector version bump — none are correctness issues

Type of Change

  • Bug fix / hardening

Testing

  • Verified every finding against official Helm chart best-practices docs and Kubernetes Pod Security Standards docs before fixing
  • helm lint --strict and helm template against the base chart, every example overlay, and multiple hand-built ESO scenarios (positive and negative)
  • Chart's own helm unittest suite: 65 → 79 tests, all passing; all 14 new tests confirmed to fail against the pre-fix code (reverted fixes, re-ran, confirmed failure, restored)
  • Confirmed the copilot label refactor produces byte-identical rendered output before/after (selector immutability risk explicitly checked, not assumed)

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…ices audit

Verified every finding against the official Helm docs and Kubernetes Pod
Security Standards docs before fixing, and validated each fix with
helm lint/template plus the chart's own helm-unittest suite (65 -> 79
tests, all new tests confirmed to fail on the pre-fix code):

- Blocker: values.schema.json documented "minimum 32/8 characters" on
  BETTER_AUTH_SECRET/ENCRYPTION_KEY/postgresql.auth.password but never
  enforced it. Added anyOf minLength-or-empty constraints (empty stays
  legal for existingSecret/ESO modes) — verified negative/positive cases
  live, no regression for any secret-delivery mode.
- Real gap: copilot didn't support the External Secrets Operator mode the
  rest of the chart offers (app/postgresql/externalDatabase). Added
  external-secret-copilot.yaml, remoteRefs.copilot, and extended
  sim.copilot.validate with the same "map it or remove it" fail-fast
  guard app.env/realtime.env already have. Verified byte-identical
  rendering for the existing non-ESO path.
- Real gap: the OpenTelemetry Collector was the only workload missing the
  shared Restricted-profile securityContext helpers (no container-level
  hardening at all). Wired sim.podSecurityContext/containerSecurityContext
  in, preserving the collector's original UID/GID/fsGroup.
- Real gap: copilot templates hand-rolled label/selector blocks instead of
  using the chart's established sim.<component>.labels/selectorLabels
  pattern. Added sim.copilot.*/sim.copilotPostgresql.* helpers and
  refactored every consumer — confirmed byte-identical helm template
  output before/after (selector labels are immutable on upgrade, so this
  was verified, not assumed).
- Documented (README): the ingressFrom default and readOnlyRootFilesystem
  posture, both real but intentional tradeoffs the audit flagged as
  underdocumented. Added extraVolumes/extraVolumeMounts to copilot's
  Deployment (realtime/pii already had it) so the readOnlyRootFilesystem
  guidance is actually actionable for all three stateless services.

Deferred (nice-to-have, not blocking): pinning the two floating Postgres
image tags, values.schema.json stubs for ~13 uncovered top-level sections,
and an OTel collector image version bump — none are correctness issues.
@vercel

vercel Bot commented Jul 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Jul 10, 2026 2:34am

Request Review

@cursor

cursor Bot commented Jul 10, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes affect secret provisioning paths (inline, existingSecret, ESO) and Copilot validation at template time; misconfigured upgrades could fail helm template/install until remoteRefs are fixed, but behavior is fail-fast rather than silent runtime breakage.

Overview
Hardens the Sim Helm chart around secret validation, Copilot + ESO parity, and Pod Security Standards, with expanded docs and regression tests.

values.schema.json now enforces minimum lengths for BETTER_AUTH_SECRET, ENCRYPTION_KEY, and postgresql.auth.password via anyOf (min length or empty string), so short inline secrets fail at install time while existingSecret/ESO can still leave values blank.

Copilot gains the same secret routing model as app: new external-secret-copilot.yaml, externalSecrets.remoteRefs.copilot, stricter sim.copilot.validate (required keys and any non-empty copilot.server.env must be mapped under ESO), and copilot.server.envDefaults for non-secret tunables (PORT, LOG_LEVEL, etc.) so ESO users do not have to map operational keys. Chart-managed copilot Secrets are skipped when ESO is on; checksum/secret hashes both inline Secret and ExternalSecret manifests to trigger rollouts on mapping changes. Copilot templates use shared sim.copilot.* label helpers, and the Deployment supports extraVolumes / extraVolumeMounts.

Telemetry OTel collector pod/container security contexts are wired through sim.podSecurityContext / sim.containerSecurityContext (Restricted defaults, original UID/GID/fsGroup preserved).

README documents default permissive networkPolicy.ingressFrom, PSS/restricted posture, and optional readOnlyRootFilesystem + /tmp emptyDir guidance. Example values and helm unittest suites are updated (including schema length and copilot secret-mode tests).

Reviewed by Cursor Bugbot for commit fd364c2. Configure here.

@greptile-apps

greptile-apps Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR hardens the Helm chart and expands copilot secret handling. The main changes are:

  • Added copilot ExternalSecret support and remoteRef validation.
  • Split copilot static defaults out of the Secret-backed env block.
  • Avoided inline default env values in copilot existingSecret mode.
  • Added Restricted Pod Security settings for telemetry.
  • Enforced documented secret-length rules in the values schema.
  • Updated chart docs and Helm unit tests for the new modes.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
helm/sim/templates/deployment-copilot.yaml Updates copilot labels, rollout checksums, env default rendering, and extra volume support.
helm/sim/templates/_helpers.tpl Adds copilot label helpers and validates copilot secret mappings for External Secrets mode.
helm/sim/templates/external-secret-copilot.yaml Adds the ExternalSecret manifest for copilot server credentials.
helm/sim/values.schema.json Adds schema checks for documented secret length requirements while keeping empty external-secret values valid.
helm/sim/tests/copilot-secret-modes_test.yaml Adds tests for copilot inline, existingSecret, and External Secrets modes.

Reviews (5): Last reviewed commit: "fix(helm): checksum/secret annotation on..." | Re-trigger Greptile

Comment thread helm/sim/templates/_helpers.tpl
Greptile caught a real bug: copilot.server.env shipped with non-empty
static defaults (PORT, SERVICE_NAME, ENVIRONMENT, LOG_LEVEL), unlike
app.env/realtime.env which ship fully empty. The new ESO validation
correctly required every non-empty env key to be mapped in
externalSecrets.remoteRefs.copilot — but that meant a default install
with copilot + ESO enabled failed demanding secret-store paths for
values that were never secrets.

Fixed by applying the chart's own existing pattern for this exact
problem: moved the 4 static keys into copilot.server.envDefaults
(mirroring app.envDefaults) and inlined them as plain container env,
bypassing the Secret/ExternalSecret system entirely — same rationale
already documented for app.envDefaults. Verified live that Greptile's
exact repro (default copilot env + ESO enabled, only the 7 real secrets
mapped) now renders cleanly and the four values still reach the
container. Added a regression test that fails on the pre-fix code.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

Comment thread helm/sim/templates/deployment-copilot.yaml
Comment thread helm/sim/templates/deployment-copilot.yaml
Greptile and Cursor Bugbot both independently caught this: in
copilot.server.secret.create=false (existingSecret) mode, the chart
still unconditionally inlined copilot.server.envDefaults as explicit
container env. Kubernetes gives explicit env precedence over envFrom,
so a pre-existing Secret's PORT/LOG_LEVEL/etc values were silently
overridden by the chart defaults — the exact shadowing bug
app.envDefaults already guards against via its own $useExistingSecret
skip, which I forgot to mirror when copying the pattern to copilot.

Skip envDefaults entirely in existingSecret mode (matching app's
existing behavior — the pre-created Secret is the sole source of
truth), while still rendering extraEnv. Verified live: existingSecret
mode now renders no env: block at all when extraEnv is unset, and
still renders extraEnv without envDefaults leaking in when it is set.
Added two regression tests, confirmed both fail on the pre-fix code.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 27d7743. Configure here.

Comment thread helm/sim/templates/deployment-copilot.yaml Outdated
…te, not the global ESO flag

Round 2's fix (which I copied nearly verbatim from Greptile's own
suggested diff) used $useExistingSecret := and (not
externalSecrets.enabled) (not copilot.server.secret.create) — Greptile
caught its own suggestion's remaining bug on round 3: when
externalSecrets.enabled=true globally (for app/postgresql) but copilot
itself uses copilot.server.secret.create=false with its own
pre-created Secret, that condition evaluated to non-existingSecret mode,
so envDefaults still inlined and shadowed the user's Secret values —
same bug, different trigger condition.

envFrom always points at the user-provided Secret name whenever
secret.create=false, independent of what other components do with ESO,
so the check should key on that alone. Verified live: global ESO
enabled + copilot's own existingSecret now renders no env: block and
envFrom correctly points at the pre-created secret name; the two
scenarios that should still inline (copilot itself on ESO, plain
inline mode) still work. Added a regression test, confirmed it fails
against round 2's guard.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

Comment thread helm/sim/templates/deployment-copilot.yaml
…secret

Cursor Bugbot caught a real bug: checksum/secret only hashed
secrets-copilot.yaml's rendered output, but under
externalSecrets.enabled=true that template renders nothing (env
credentials come from external-secret-copilot.yaml instead). Result:
changing externalSecrets.remoteRefs.copilot mappings wouldn't change
the pod template hash, so Kubernetes would never restart the copilot
pod to pick up the new mapping — stale envFrom values until a manual
restart.

Fixed by hashing the concatenation of both templates' rendered output:
whichever mode is active, only one renders non-empty content, but the
concatenated hash still changes on a mode switch or a remoteRefs
change. This can't reach into the live secret store value ESO syncs
(Helm only sees the ExternalSecret manifest at render time) — that's
an inherent ESO limitation, not something a checksum annotation can
close; documented as such in the template comment.

Note: deployment-app.yaml and deployment-realtime.yaml have this same
latent limitation in ESO mode (checksum/secret only hashes
secrets-app.yaml), but that's pre-existing code outside this PR's
diff — not fixed here to stay scoped to what Cursor actually flagged.

Verified live: the checksum differs across two different
remoteRefs.copilot.LICENSE_KEY mappings, and still changes correctly
in plain inline mode. Added a regression test.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit fd364c2. Configure here.

@waleedlatif1 waleedlatif1 merged commit 896d15b into staging Jul 10, 2026
18 checks passed
@waleedlatif1 waleedlatif1 deleted the helm-best-practices-hardening branch July 10, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant