-
Notifications
You must be signed in to change notification settings - Fork 4
feat(cilium): stage bandwidth manager with BBR #2608
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
devantler
wants to merge
4
commits into
main
Choose a base branch
from
codex/cilium-bbr-stage-2607
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
a3880bb
feat(cilium): stage bandwidth manager with BBR
devantler d8c9c59
fix(cilium): guard bandwidth manager activation
devantler 0a1f812
test(cilium): replace Python BBR test harness
devantler 5719b84
test(cilium): drain render stream in extractor
devantler File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
51 changes: 51 additions & 0 deletions
51
...ner/infrastructure/controllers/cilium/components/bandwidth-manager-bbr/kustomization.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| --- | ||
| # Default-off staging component for platform#2029 / platform#2607. | ||
| # | ||
| # Cilium's bandwidth manager installs fq qdiscs and EDT pacing, while BBR | ||
| # selects BBR congestion control for eligible egress traffic. The Talos 6.x | ||
| # kernel is newer than Cilium's Linux 5.18 minimum for BBR. This component is | ||
| # Hetzner-only because that is the production path where egress traverses the | ||
| # node datapath and where WireGuard is enabled. | ||
| # | ||
| # HOST-NAMESPACE GUARD: pod BBR also requires eBPF host routing. The current | ||
| # tunnel-mode datapath intentionally keeps bpf.masquerade off, so | ||
| # bbrHostNamespaceOnly stays true and limits BBR to hostNetwork workloads. | ||
| # Do not remove that guard until #2609 has merged and completed its own soak. | ||
| # | ||
| # DEVICE GUARD: production currently pins two heterogeneous private-NIC names | ||
| # (`enp7s0` on workers, `eth1` on control planes). Cilium requires every | ||
| # manually listed bandwidth-manager device name on every managed node. Do not | ||
| # activate this component until #2610 provides a homogeneous selection while | ||
| # preserving the private-NIC WireGuard and mutual-auth path. | ||
| # | ||
| # SAFETY: this component is staged, not activation-ready. Keep it unreferenced | ||
| # until both #2609 and #2610 are complete, then use a separate activation/soak. | ||
| # `rollOutCiliumPods: true` means enabling it rolls every Cilium agent one node | ||
| # at a time and replaces the node qdisc. Activate only in a low-traffic window; | ||
| # verify `cilium status --verbose` and unexpected Hubble drops after the roll. | ||
| # Roll back by removing the component reference, which restores the chart | ||
| # defaults in the next one-node-at-a-time roll. | ||
| # | ||
| # bpf.masquerade and netkit are deliberately out of scope and require their own | ||
| # independently validated slices. | ||
| apiVersion: kustomize.config.k8s.io/v1alpha1 | ||
| kind: Component | ||
| patches: | ||
| - target: | ||
| group: helm.toolkit.fluxcd.io | ||
| version: v2 | ||
| kind: HelmRelease | ||
| name: cilium | ||
| namespace: kube-system | ||
| patch: | | ||
| apiVersion: helm.toolkit.fluxcd.io/v2 | ||
| kind: HelmRelease | ||
| metadata: | ||
| name: cilium | ||
| namespace: kube-system | ||
| spec: | ||
| values: | ||
| bandwidthManager: | ||
| enabled: true | ||
| bbr: true | ||
|
devantler marked this conversation as resolved.
|
||
| bbrHostNamespaceOnly: true | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
127 changes: 127 additions & 0 deletions
127
scripts/tests/test-cilium-bandwidth-manager-component.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" | ||
| readonly root_dir | ||
| readonly controllers_dir="${root_dir}/k8s/providers/hetzner/infrastructure/controllers" | ||
| readonly opt_in_fixture="${root_dir}/tests/cilium-bandwidth-manager-bbr" | ||
|
|
||
| fail() { | ||
| printf 'FAIL: %s\n' "$1" >&2 | ||
| exit 1 | ||
| } | ||
|
|
||
| extract_cilium_release() { | ||
| awk ' | ||
| function reset_document() { | ||
| document = "" | ||
| is_helm_release = 0 | ||
| is_cilium = 0 | ||
| } | ||
|
|
||
| function emit_if_cilium() { | ||
| if (!found && is_helm_release && is_cilium) { | ||
| printf "%s", document | ||
| found = 1 | ||
| } | ||
| reset_document() | ||
| } | ||
|
|
||
| BEGIN { reset_document() } | ||
| /^---[[:space:]]*$/ { emit_if_cilium(); next } | ||
| { | ||
| if (!found) { | ||
| document = document $0 ORS | ||
| if ($0 ~ /^kind:[[:space:]]*HelmRelease[[:space:]]*$/) { | ||
| is_helm_release = 1 | ||
| } | ||
| if ($0 ~ /^ name:[[:space:]]*cilium[[:space:]]*$/) { | ||
| is_cilium = 1 | ||
| } | ||
| } | ||
| } | ||
| END { | ||
| if (!found) { | ||
| emit_if_cilium() | ||
| } | ||
| if (!found) { | ||
| exit 1 | ||
| } | ||
| } | ||
| ' | ||
| } | ||
|
|
||
| require_text() { | ||
| local haystack="$1" | ||
| local needle="$2" | ||
| local description="$3" | ||
|
|
||
| grep -Fq -- "$needle" <<<"${haystack}" || fail "${description}" | ||
| } | ||
|
|
||
| reject_text() { | ||
| local haystack="$1" | ||
| local needle="$2" | ||
| local description="$3" | ||
|
|
||
| if grep -Fq -- "$needle" <<<"${haystack}"; then | ||
| fail "${description}" | ||
| fi | ||
| } | ||
|
|
||
| extractor_probe="$( | ||
| { | ||
| printf '%s\n' \ | ||
| 'apiVersion: helm.toolkit.fluxcd.io/v2' \ | ||
| 'kind: HelmRelease' \ | ||
| 'metadata:' \ | ||
| ' name: cilium' \ | ||
| '---' | ||
| for ((line = 0; line < 10000; line++)); do | ||
| printf '# trailing render content %05d\n' "${line}" | ||
| done | ||
| } | extract_cilium_release | ||
| )" || fail 'the Cilium release extractor must consume the complete render stream' | ||
| require_text \ | ||
| "${extractor_probe}" \ | ||
| 'kind: HelmRelease' \ | ||
| 'the Cilium release extractor must return the matching document' | ||
|
|
||
| readonly controllers_kustomization="${controllers_dir}/kustomization.yaml" | ||
| grep -Fxq ' # - cilium/components/bandwidth-manager-bbr/' "${controllers_kustomization}" || | ||
| fail 'the production controllers overlay must retain the documented opt-in reference' | ||
| if grep -Fxq ' - cilium/components/bandwidth-manager-bbr/' "${controllers_kustomization}"; then | ||
| fail 'the production controllers overlay must keep the BBR component disabled by default' | ||
| fi | ||
|
|
||
| default_release="$(kubectl kustomize "${controllers_dir}" | extract_cilium_release)" || | ||
|
devantler marked this conversation as resolved.
|
||
| fail 'the default production controllers render has no Cilium HelmRelease' | ||
| reject_text \ | ||
| "${default_release}" \ | ||
| 'bandwidthManager:' \ | ||
| 'the default production render unexpectedly enables the bandwidth manager' | ||
|
|
||
| opt_in_release="$( | ||
| kubectl kustomize "${opt_in_fixture}" --load-restrictor LoadRestrictionsNone | | ||
| extract_cilium_release | ||
| )" || fail 'the opt-in fixture render has no Cilium HelmRelease' | ||
|
|
||
| require_text \ | ||
| "${opt_in_release}" \ | ||
| $'bandwidthManager:\n bbr: true\n bbrHostNamespaceOnly: true\n enabled: true' \ | ||
| 'the opt-in render must enable host-namespace-only BBR' | ||
| reject_text \ | ||
| "${opt_in_release}" \ | ||
| $'bpf:\n masquerade: true' \ | ||
| 'the opt-in render must not enable BPF masquerading' | ||
| require_text \ | ||
| "${opt_in_release}" \ | ||
| $'encryption:\n enabled: true\n nodeEncryption: false' \ | ||
| 'the opt-in render must preserve the production encryption settings' | ||
| require_text \ | ||
| "${opt_in_release}" \ | ||
| 'type: wireguard' \ | ||
| 'the opt-in render must preserve WireGuard encryption' | ||
|
|
||
| printf 'PASS: Cilium bandwidth manager is default-off and the opt-in render preserves production guards\n' | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| resources: | ||
| - ../../k8s/providers/hetzner/infrastructure/controllers/ | ||
| components: | ||
| - ../../k8s/providers/hetzner/infrastructure/controllers/cilium/components/bandwidth-manager-bbr/ |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.