feat(Kind): add AddManifest resource for kubectl apply after cluster ready - #1481
feat(Kind): add AddManifest resource for kubectl apply after cluster ready#1481tamirdresher wants to merge 16 commits into
Conversation
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.sh | bash -s -- 1481Or
iex "& { $(irm https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.ps1) } 1481" |
5180420 to
d708bd3
Compare
…ready
Adds Kubernetes manifest resources for Kind clusters, aligned with the existing K3s manifest pattern while keeping Kind's host-side kubectl execution model.
Public API surface:
cluster.AddManifest(name, manifestPath)
.WithNamespace(ns)
.WithRecursive()
.WithServerSideApply(forceConflicts: false)
.WithFieldManager("aspire-apphost")
.WithApplyTimeout(TimeSpan.FromMinutes(5))
.WithCrdWaitTimeout(TimeSpan.FromMinutes(5))
.WithCrdWaitBehavior(CrdWaitBehavior.Fail)
cluster.AddManifestFromContent(name, content)
.WithNamespace(ns)
.WithServerSideApply(forceConflicts: false)
.WithFieldManager("aspire-apphost")
Key behavior:
- K8sManifestResource extends KindDeployedResource(name, parent) and is exported for Aspire tooling.
- AddManifest resolves relative paths against AppHostDirectory; Kustomize detection happens at apply time and supports case-insensitive marker variants.
- AddManifestFromContent applies inline YAML via kubectl apply -f - using stdin, avoiding temporary files.
- KubectlManager probes kubectl cluster-info before apply, bounds apply with ApplyTimeout, and fails CRD waits by default with an opt-in best-effort mode.
- WithRecursive is honored for file/directory applies and ignored with a warning for Kustomize or stdin applies.
- Manifest resources rely on Running/FailedToStart state rather than a default empty-selector workload health check.
Downstream resources can WaitFor the manifest resource so they only start once kubectl apply succeeds:
var crds = cluster.AddManifest("crds", "./manifests/crds.yaml");
builder.AddContainer("my-operator", "my-org/operator")
.WithReference(cluster)
.WaitFor(crds);
Manifests persist with the cluster - deleted with session clusters, retained with persistent clusters.
Tests cover resource registration, parent wiring, null guards, AppHost-relative path resolution, absolute-path preservation, Kustomize detection, stdin apply argument shape and input piping, recursive ignore behavior, cluster-info retry, apply timeout cancellation, CRD wait argument shape, and default/best-effort CRD wait behavior.
Closes: none (net new)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7c8c8109-2ae6-407b-8740-8c3018abe8f1
d708bd3 to
5c80dc4
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add opt-in CRD wait retries and --set-string support for Helm chart values. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add exported node image and host mount APIs and trigger session cluster cleanup on application stopping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add best-effort process-exit and Ctrl+C cleanup on top of ApplicationStopping, and cover standalone stop-trigger cleanup plus idempotence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve relative node mount paths against the AppHost directory and make Helm value APIs last-write-wins across --set and --set-string. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Also clarify that the node-mount sample is a configuration example rather than an end-to-end in-cluster mount consumer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the generated Kind API reference source expected by the repo's GenAPI workflow and apply the repo's formatting conventions to the touched sample/source/test files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add fuller XML documentation for the new public Kind fluent APIs so their timeout, CRD retry, Helm string-value, node image, and node mount semantics match the surrounding source documentation style. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR extends CommunityToolkit.Aspire.Hosting.Kind with a new run-mode resource (K8sManifestResource) and fluent APIs to apply Kubernetes YAML (file/dir/kustomize/inline) via kubectl apply after a Kind cluster becomes reachable, enabling downstream resources to WaitFor(...) successful applies. It also enhances the Kind integration with node image + node mount configuration, improves Helm chart value handling (--set-string) and adds a Helm CRD-race retry mechanism.
Changes:
- Add
AddManifest/AddManifestFromContentand supportingKubectlManagerorchestration (cluster-info probe, apply args, CRD Established waits). - Add Kind cluster configuration enhancements (
WithNodeImage,WithNodeMount) and improve shutdown cleanup signaling. - Expand Kind test suite and update README + example AppHost to document and demonstrate the new APIs.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/Directory.Build.props | Enables dotnet test support for Microsoft.Testing.Platform projects. |
| tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/WithKindTests.cs | Adds assertions for node image + node mounts fluent APIs. |
| tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/KindPublicApiTests.cs | Adds null-guard tests for new Kind fluent APIs. |
| tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/KindManifestTests.cs | Adds comprehensive tests for manifest resource + kubectl argument shaping/behavior. |
| tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/KindHelmChartTests.cs | Adds coverage for --set-string and Helm CRD wait retry behavior. |
| tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/FakeProcessRunner.cs | Adds delay + stdin support to process runner test double. |
| tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/AddKindClusterTests.cs | Adds tests for node image/mount, lifecycle cleanup behavior, and kubeconfig redaction. |
| src/CommunityToolkit.Aspire.Hosting.Kind/README.md | Documents new manifest apply APIs, new Helm behaviors, and node config options. |
| src/CommunityToolkit.Aspire.Hosting.Kind/KubectlTimeouts.cs | Centralizes timeout normalization and bounds for kubectl-related waits. |
| src/CommunityToolkit.Aspire.Hosting.Kind/KubectlManager.cs | Implements kubectl orchestration: cluster-info probe, apply, CRD waits, arg builders. |
| src/CommunityToolkit.Aspire.Hosting.Kind/KindManifestResourceBuilderExtensions.cs | Adds public AddManifest* and fluent configuration methods + lifecycle hook wiring. |
| src/CommunityToolkit.Aspire.Hosting.Kind/KindHelmChartResourceBuilderExtensions.cs | Adds WithHelmStringValue and WithCrdWaitRetry, and clarifies namespace behavior docs. |
| src/CommunityToolkit.Aspire.Hosting.Kind/KindHelmChartResource.cs | Adds StringValues and CRD retry configuration fields for Helm resource. |
| src/CommunityToolkit.Aspire.Hosting.Kind/KindConfigGenerator.cs | Applies fully-qualified node images and extra mounts to generated Kind config. |
| src/CommunityToolkit.Aspire.Hosting.Kind/KindConfigAnnotation.cs | Adds annotations for full node image override and node mounts. |
| src/CommunityToolkit.Aspire.Hosting.Kind/KindClusterResourceBuilderExtensions.cs | Adds WithNodeImage and WithNodeMount exported APIs. |
| src/CommunityToolkit.Aspire.Hosting.Kind/KindClusterLifecycleHook.cs | Registers additional shutdown/process-exit handlers for best-effort cleanup. |
| src/CommunityToolkit.Aspire.Hosting.Kind/K8sManifestResource.cs | Introduces new resource type representing an applied manifest. |
| src/CommunityToolkit.Aspire.Hosting.Kind/IProcessRunner.cs | Extends process runner API to support passing stdin content. |
| src/CommunityToolkit.Aspire.Hosting.Kind/HelmManager.cs | Adds Helm CRD-race detection, CRD snapshotting, and retry with backoff. |
| src/CommunityToolkit.Aspire.Hosting.Kind/DefaultProcessRunner.cs | Adds stdin piping and kubeconfig path redaction in debug logging. |
| src/CommunityToolkit.Aspire.Hosting.Kind/CrdWaitBehavior.cs | Adds exported enum controlling CRD wait failure behavior. |
| src/CommunityToolkit.Aspire.Hosting.Kind/api/CommunityToolkit.Aspire.Hosting.Kind.cs | Updates API baseline to reflect new exported APIs/types. |
| examples/kind/CommunityToolkit.Aspire.Hosting.Kind.AppHost/Program.cs | Demonstrates node mounts, manifests apply, SSA, and waiting on manifest resources. |
| examples/kind/CommunityToolkit.Aspire.Hosting.Kind.AppHost/manifests/extra-config.yaml | Adds example multi-document Kubernetes manifest used by the sample. |
| examples/kind/CommunityToolkit.Aspire.Hosting.Kind.AppHost/CommunityToolkit.Aspire.Hosting.Kind.AppHost.csproj | Copies manifest files to output for the sample AppHost. |
Comments suppressed due to low confidence (2)
src/CommunityToolkit.Aspire.Hosting.Kind/KubectlManager.cs:268
- CreateClusterInfoArguments should reject empty kubeconfigPath values up front (not just null) to avoid producing a kubectl command that will certainly fail.
internal static IReadOnlyList<string> CreateClusterInfoArguments(string kubeconfigPath)
{
ArgumentNullException.ThrowIfNull(kubeconfigPath);
src/CommunityToolkit.Aspire.Hosting.Kind/KubectlManager.cs:279
- CreateGetCrdsArguments should reject empty kubeconfigPath values up front (not just null) to avoid producing a kubectl command that will certainly fail.
internal static IReadOnlyList<string> CreateGetCrdsArguments(string kubeconfigPath)
{
ArgumentNullException.ThrowIfNull(kubeconfigPath);
| internal static TimeSpan ComputeRetryBackoff(TimeSpan initialBackoff, int failureCount) | ||
| { | ||
| ArgumentOutOfRangeException.ThrowIfLessThan(failureCount, 1); | ||
|
|
||
| var multiplier = 1 << (failureCount - 1); | ||
| var scaledTicks = initialBackoff.Ticks * multiplier; | ||
| return scaledTicks >= TimeSpan.MaxValue.Ticks | ||
| ? TimeSpan.MaxValue | ||
| : TimeSpan.FromTicks(scaledTicks); | ||
| } |
| if (result.ExitCode != 0) | ||
| { | ||
| throw new InvalidOperationException( | ||
| $"Failed to apply manifest '{resource.ManifestPath}' to cluster '{resource.Parent.Name}': {result.Error}"); | ||
| } |
| ArgumentNullException.ThrowIfNull(crdNames); | ||
| ArgumentNullException.ThrowIfNull(kubeconfigPath); | ||
|
|
| ArgumentNullException.ThrowIfNull(builder); | ||
| ArgumentException.ThrowIfNullOrWhiteSpace(hostPath); | ||
| ArgumentException.ThrowIfNullOrWhiteSpace(containerPath); |
| ArgumentNullException.ThrowIfNull(builder); | ||
| ArgumentNullException.ThrowIfNull(name); | ||
| ArgumentNullException.ThrowIfNull(manifestPath); | ||
|
|
| var processRunner = e.Services.GetRequiredService<IProcessRunner>(); | ||
| var kubectlManager = CreateKubectlManager(processRunner, resource); | ||
| await kubectlManager.ApplyAsync(resource, logger, ct); | ||
|
|
| private void RunCleanupSynchronously() | ||
| { | ||
| try | ||
| { | ||
| EnsureCleanupStarted().GetAwaiter().GetResult(); | ||
| } | ||
| catch | ||
| { | ||
| } | ||
| } |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Motivation
For dev-loop scenarios that run against a Kind cluster during F5 (operators, controllers, admission webhooks, or in-source contributions to an OSS Kubernetes project) the AppHost often needs to apply a set of
CustomResourceDefinition,Namespace,ServiceAccount,ClusterRole,ClusterRoleBinding, orConfigMapobjects to the cluster after the cluster reaches Running but before dependent resources start.Today
CommunityToolkit.Aspire.Hosting.Kindhandles the Helm side of this viacluster.AddHelmChart(...), but if you just have static YAML you have to hand-roll a lifecycle hook in the AppHost. This PR adds the natural equivalent ofAddHelmChartfor thekubectl applycase.This adds runtime manifest apply (at F5 /
aspire start). It is distinct fromAspire.Hosting.Kubernetes'sPublishAsKubernetesService(r => r.AdditionalResources.Add(...)), which is build-time (baked into a generated Helm chart ataspire publish). Both have legitimate use cases; this PR fills the runtime gap for local Kind clusters.What this adds
A new resource + extension API surface, all public APIs exported for Aspire tooling:
Design
Mirrors the existing Kind deployed-resource pattern and K3s manifest semantics while keeping Kind's host-side kubectl execution model:
KindHelmChartResourceK8sManifestResource : KindDeployedResourceHelmManagerKubectlManager(wrapsIProcessRunner, internal static argument builders for testability)KindHelmChartResourceBuilderExtensionsKindManifestResourceBuilderExtensionsLifecycle callback matches
AddHelmChartstep-for-step:WaitForResourceAsync(parent.Name, Running)PublishAsync(BeforeResourceStartedEvent)PublishUpdateAsync(Starting)KubectlManager.ApplyAsync->PublishUpdateAsync(Running)}PublishUpdateAsync(FailedToStart)-> rethrow }KubectlManager.ApplyAsyncnow probeskubectl cluster-infobefore apply so the Kind resource state and Kubernetes API reachability are both satisfied before manifests are applied.New public API
AddManifest(name, manifestPath)kubectl apply -f <path> --kubeconfig=<cluster kubeconfig>orkubectl apply -k <path>for Kustomize directoriesAddManifestFromContent(name, content)kubectl apply -f - --kubeconfig=<cluster kubeconfig>with content on stdinWithNamespace(ns)(inherited fromKindDeployedResource)--namespace <ns>(does not create the namespace for manifests)WithRecursive()--recursivefor normal file/directory applies; ignored with a warning for Kustomize/stdinWithServerSideApply(forceConflicts: false)--server-side(+--force-conflicts)WithFieldManager(name)--field-manager <name>WithApplyTimeout(timeout)kubectl applyexecution time (default 5 minutes)WithCrdWaitTimeout(timeout)Establishedwait timeout (default 5 minutes)WithCrdWaitBehavior(behavior)Failby default;BestEffortlogs a warning and continuesTests
xUnit v3 coverage in
KindManifestTests.cs/AddKindClusterTests.csnow includes:WithRecursive,WithServerSideApply(forceConflicts),WithFieldManager,WithApplyTimeout,WithCrdWaitTimeout, andWithCrdWaitBehaviorKubectlManager.CreateApplyArguments,CreateWaitArguments, and cluster-info retry behaviorFull
CommunityToolkit.Aspire.Hosting.Kind.Testsproject passes locally: 206 / 206.Real-world usage
Upstreamed from https://github.com/tamirdresher/aspire-argocd-dev-loop, which spins up an argo-cd F5 inner loop against a Kind cluster and uses
AddManifestto install the ArgoCD manifests before theargocd-serverandrepo-serverprojects start. Prior to this API the sample carried a hand-rolledOnBeforeStarthook with its ownProcess.Start("kubectl", ...).Open questions for reviewers
Deferred: naming consistency between this integration's
AddManifestandCommunityToolkit.Aspire.Hosting.K3s'sAddK8sManifest. Both are[AspireExport]-ed public API in separate packages; unifying would be a breaking change for K3s consumers. Happy to align on either name if maintainers have a preference.Follow-up
I have a separate small PR queued:
WithPortMapping(hostPort, containerPort)as a fluent shorthand over the existingWithKindConfig(Action<KindConfigModel>)for exposing NodePort services on127.0.0.1. Shipping it separately to keep this PR focused on the new resource type.Docs
README section updated in
src/CommunityToolkit.Aspire.Hosting.Kind/README.md: usage examples for single file, recursive directory, Kustomize auto-detect, inline content, server-side apply, CRD wait behavior, namespace behavior, security notes, and theAPI referencetable entries.Checklist
KindHelmChartResourceand K3sK8sManifestResourcesemantics)[AspireExport]Follow-on docs
learn.microsoft.com/dotnet/aspire/community-toolkit/hosting-kindlives inMicrosoftDocs/aspire-docs, outside this fork's tree, and will need a matching update forAddManifest. That can be handled as a separate PR after this merges.