Skip to content

feat(Kind): add AddManifest resource for kubectl apply after cluster ready - #1481

Open
tamirdresher wants to merge 16 commits into
CommunityToolkit:mainfrom
tamirdresher:feat/kind-with-manifest
Open

feat(Kind): add AddManifest resource for kubectl apply after cluster ready#1481
tamirdresher wants to merge 16 commits into
CommunityToolkit:mainfrom
tamirdresher:feat/kind-with-manifest

Conversation

@tamirdresher

@tamirdresher tamirdresher commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Motivation

For dev-loop scenarios that run against a Kind cluster during F5 (operators, controllers, admission webhooks, or in-source contributions to an OSS Kubernetes project) the AppHost often needs to apply a set of CustomResourceDefinition, Namespace, ServiceAccount, ClusterRole, ClusterRoleBinding, or ConfigMap objects to the cluster after the cluster reaches Running but before dependent resources start.

Today CommunityToolkit.Aspire.Hosting.Kind handles the Helm side of this via cluster.AddHelmChart(...), but if you just have static YAML you have to hand-roll a lifecycle hook in the AppHost. This PR adds the natural equivalent of AddHelmChart for the kubectl apply case.

This adds runtime manifest apply (at F5 / aspire start). It is distinct from Aspire.Hosting.Kubernetes's PublishAsKubernetesService(r => r.AdditionalResources.Add(...)), which is build-time (baked into a generated Helm chart at aspire publish). Both have legitimate use cases; this PR fills the runtime gap for local Kind clusters.

What this adds

A new resource + extension API surface, all public APIs exported for Aspire tooling:

var cluster = builder.AddKindCluster("mycluster");

// Single file
cluster.AddManifest("crds", "./manifests/crds.yaml");

// Directory, recursive, into a namespace
cluster.AddManifest("platform", "./manifests/platform")
    .WithRecursive()
    .WithNamespace("platform");

// Kustomize overlay directory: detected at apply time and applied with kubectl apply -k
cluster.AddManifest("overlay", "./manifests/overlays/dev");

// Server-side apply with a stable field manager
cluster.AddManifest("operator", "./manifests/argocd/install.yaml")
    .WithServerSideApply(forceConflicts: true)
    .WithFieldManager("aspire-apphost");

// Inline content, applied through stdin without temp files
cluster.AddManifestFromContent("demo-ns", """
    apiVersion: v1
    kind: Namespace
    metadata:
      name: aspire-demo
    """);

// Downstream resources can wait for the apply to succeed
var crds = cluster.AddManifest("crds", "./manifests/crds.yaml");

builder.AddContainer("my-operator", "my-org/operator")
    .WithReference(cluster)
    .WaitFor(crds);

Design

Mirrors the existing Kind deployed-resource pattern and K3s manifest semantics while keeping Kind's host-side kubectl execution model:

Layer Helm equivalent New file
Resource KindHelmChartResource K8sManifestResource : KindDeployedResource
CLI orchestration HelmManager KubectlManager (wraps IProcessRunner, internal static argument builders for testability)
Extensions KindHelmChartResourceBuilderExtensions KindManifestResourceBuilderExtensions

Lifecycle callback matches AddHelmChart step-for-step:

  1. WaitForResourceAsync(parent.Name, Running)
  2. PublishAsync(BeforeResourceStartedEvent)
  3. PublishUpdateAsync(Starting)
  4. try { KubectlManager.ApplyAsync -> PublishUpdateAsync(Running) }
  5. catch { PublishUpdateAsync(FailedToStart) -> rethrow }

KubectlManager.ApplyAsync now probes kubectl cluster-info before apply so the Kind resource state and Kubernetes API reachability are both satisfied before manifests are applied.

New public API

Method Maps to
AddManifest(name, manifestPath) kubectl apply -f <path> --kubeconfig=<cluster kubeconfig> or kubectl apply -k <path> for Kustomize directories
AddManifestFromContent(name, content) kubectl apply -f - --kubeconfig=<cluster kubeconfig> with content on stdin
WithNamespace(ns) (inherited from KindDeployedResource) --namespace <ns> (does not create the namespace for manifests)
WithRecursive() --recursive for normal file/directory applies; ignored with a warning for Kustomize/stdin
WithServerSideApply(forceConflicts: false) --server-side (+ --force-conflicts)
WithFieldManager(name) --field-manager <name>
WithApplyTimeout(timeout) Bounded kubectl apply execution time (default 5 minutes)
WithCrdWaitTimeout(timeout) CRD Established wait timeout (default 5 minutes)
WithCrdWaitBehavior(behavior) Fail by default; BestEffort logs a warning and continues

Tests

xUnit v3 coverage in KindManifestTests.cs / AddKindClusterTests.cs now includes:

  • Resource creation, parent wiring, name/path behavior, and inline content registration
  • Null-guard tests for builder, name, manifestPath/content, and fieldManager
  • AppHost-relative path resolution and absolute-path preservation
  • Kustomize detection for lowercase and capitalized marker variants, including apply-time detection
  • Property-set tests for WithRecursive, WithServerSideApply(forceConflicts), WithFieldManager, WithApplyTimeout, WithCrdWaitTimeout, and WithCrdWaitBehavior
  • Default-value tests for recursive/server-side/force-conflicts, field manager, namespace, apply timeout, and CRD wait behavior
  • KubectlManager.CreateApplyArguments, CreateWaitArguments, and cluster-info retry behavior
  • Apply timeout cancellation
  • Default fail-fast and opt-in best-effort CRD wait behavior
  • kubeconfig path redaction in debug logs

Full CommunityToolkit.Aspire.Hosting.Kind.Tests project passes locally: 206 / 206.

Real-world usage

Upstreamed from https://github.com/tamirdresher/aspire-argocd-dev-loop, which spins up an argo-cd F5 inner loop against a Kind cluster and uses AddManifest to install the ArgoCD manifests before the argocd-server and repo-server projects start. Prior to this API the sample carried a hand-rolled OnBeforeStart hook with its own Process.Start("kubectl", ...).

Open questions for reviewers

Deferred: naming consistency between this integration's AddManifest and CommunityToolkit.Aspire.Hosting.K3s's AddK8sManifest. Both are [AspireExport]-ed public API in separate packages; unifying would be a breaking change for K3s consumers. Happy to align on either name if maintainers have a preference.

Follow-up

I have a separate small PR queued: WithPortMapping(hostPort, containerPort) as a fluent shorthand over the existing WithKindConfig(Action<KindConfigModel>) for exposing NodePort services on 127.0.0.1. Shipping it separately to keep this PR focused on the new resource type.

Docs

README section updated in src/CommunityToolkit.Aspire.Hosting.Kind/README.md: usage examples for single file, recursive directory, Kustomize auto-detect, inline content, server-side apply, CRD wait behavior, namespace behavior, security notes, and the API reference table entries.

Checklist

  • Design mirrors existing resource patterns (KindHelmChartResource and K3s K8sManifestResource semantics)
  • Public API export surface is covered where Aspire supports [AspireExport]
  • Tests pass locally (206 / 206)
  • README updated

Follow-on docs

learn.microsoft.com/dotnet/aspire/community-toolkit/hosting-kind lives in MicrosoftDocs/aspire-docs, outside this fork's tree, and will need a matching update for AddManifest. That can be handled as a separate PR after this merges.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.sh | bash -s -- 1481

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.ps1) } 1481"

@tamirdresher
tamirdresher force-pushed the feat/kind-with-manifest branch 4 times, most recently from 5180420 to d708bd3 Compare July 27, 2026 06:06
…ready

Adds Kubernetes manifest resources for Kind clusters, aligned with the existing K3s manifest pattern while keeping Kind's host-side kubectl execution model.

Public API surface:

  cluster.AddManifest(name, manifestPath)
         .WithNamespace(ns)
         .WithRecursive()
         .WithServerSideApply(forceConflicts: false)
         .WithFieldManager("aspire-apphost")
         .WithApplyTimeout(TimeSpan.FromMinutes(5))
         .WithCrdWaitTimeout(TimeSpan.FromMinutes(5))
         .WithCrdWaitBehavior(CrdWaitBehavior.Fail)

  cluster.AddManifestFromContent(name, content)
         .WithNamespace(ns)
         .WithServerSideApply(forceConflicts: false)
         .WithFieldManager("aspire-apphost")

Key behavior:

  - K8sManifestResource extends KindDeployedResource(name, parent) and is exported for Aspire tooling.
  - AddManifest resolves relative paths against AppHostDirectory; Kustomize detection happens at apply time and supports case-insensitive marker variants.
  - AddManifestFromContent applies inline YAML via kubectl apply -f - using stdin, avoiding temporary files.
  - KubectlManager probes kubectl cluster-info before apply, bounds apply with ApplyTimeout, and fails CRD waits by default with an opt-in best-effort mode.
  - WithRecursive is honored for file/directory applies and ignored with a warning for Kustomize or stdin applies.
  - Manifest resources rely on Running/FailedToStart state rather than a default empty-selector workload health check.

Downstream resources can WaitFor the manifest resource so they only start once kubectl apply succeeds:

  var crds = cluster.AddManifest("crds", "./manifests/crds.yaml");

  builder.AddContainer("my-operator", "my-org/operator")
      .WithReference(cluster)
      .WaitFor(crds);

Manifests persist with the cluster - deleted with session clusters, retained with persistent clusters.

Tests cover resource registration, parent wiring, null guards, AppHost-relative path resolution, absolute-path preservation, Kustomize detection, stdin apply argument shape and input piping, recursive ignore behavior, cluster-info retry, apply timeout cancellation, CRD wait argument shape, and default/best-effort CRD wait behavior.

Closes: none (net new)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7c8c8109-2ae6-407b-8740-8c3018abe8f1
@tamirdresher
tamirdresher force-pushed the feat/kind-with-manifest branch from d708bd3 to 5c80dc4 Compare July 27, 2026 07:05
Copilot and others added 14 commits July 29, 2026 10:31
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add opt-in CRD wait retries and --set-string support for Helm chart values.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add exported node image and host mount APIs and trigger session cluster cleanup on application stopping.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add best-effort process-exit and Ctrl+C cleanup on top of ApplicationStopping, and cover standalone stop-trigger cleanup plus idempotence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve relative node mount paths against the AppHost directory and make Helm value APIs last-write-wins across --set and --set-string.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Also clarify that the node-mount sample is a configuration example rather than an end-to-end in-cluster mount consumer.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the generated Kind API reference source expected by the repo's GenAPI workflow and apply the repo's formatting conventions to the touched sample/source/test files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add fuller XML documentation for the new public Kind fluent APIs so their timeout, CRD retry, Helm string-value, node image, and node mount semantics match the surrounding source documentation style.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tamirdresher
tamirdresher marked this pull request as ready for review July 29, 2026 12:12
Copilot AI review requested due to automatic review settings July 29, 2026 12:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends CommunityToolkit.Aspire.Hosting.Kind with a new run-mode resource (K8sManifestResource) and fluent APIs to apply Kubernetes YAML (file/dir/kustomize/inline) via kubectl apply after a Kind cluster becomes reachable, enabling downstream resources to WaitFor(...) successful applies. It also enhances the Kind integration with node image + node mount configuration, improves Helm chart value handling (--set-string) and adds a Helm CRD-race retry mechanism.

Changes:

  • Add AddManifest / AddManifestFromContent and supporting KubectlManager orchestration (cluster-info probe, apply args, CRD Established waits).
  • Add Kind cluster configuration enhancements (WithNodeImage, WithNodeMount) and improve shutdown cleanup signaling.
  • Expand Kind test suite and update README + example AppHost to document and demonstrate the new APIs.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
tests/Directory.Build.props Enables dotnet test support for Microsoft.Testing.Platform projects.
tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/WithKindTests.cs Adds assertions for node image + node mounts fluent APIs.
tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/KindPublicApiTests.cs Adds null-guard tests for new Kind fluent APIs.
tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/KindManifestTests.cs Adds comprehensive tests for manifest resource + kubectl argument shaping/behavior.
tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/KindHelmChartTests.cs Adds coverage for --set-string and Helm CRD wait retry behavior.
tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/FakeProcessRunner.cs Adds delay + stdin support to process runner test double.
tests/CommunityToolkit.Aspire.Hosting.Kind.Tests/AddKindClusterTests.cs Adds tests for node image/mount, lifecycle cleanup behavior, and kubeconfig redaction.
src/CommunityToolkit.Aspire.Hosting.Kind/README.md Documents new manifest apply APIs, new Helm behaviors, and node config options.
src/CommunityToolkit.Aspire.Hosting.Kind/KubectlTimeouts.cs Centralizes timeout normalization and bounds for kubectl-related waits.
src/CommunityToolkit.Aspire.Hosting.Kind/KubectlManager.cs Implements kubectl orchestration: cluster-info probe, apply, CRD waits, arg builders.
src/CommunityToolkit.Aspire.Hosting.Kind/KindManifestResourceBuilderExtensions.cs Adds public AddManifest* and fluent configuration methods + lifecycle hook wiring.
src/CommunityToolkit.Aspire.Hosting.Kind/KindHelmChartResourceBuilderExtensions.cs Adds WithHelmStringValue and WithCrdWaitRetry, and clarifies namespace behavior docs.
src/CommunityToolkit.Aspire.Hosting.Kind/KindHelmChartResource.cs Adds StringValues and CRD retry configuration fields for Helm resource.
src/CommunityToolkit.Aspire.Hosting.Kind/KindConfigGenerator.cs Applies fully-qualified node images and extra mounts to generated Kind config.
src/CommunityToolkit.Aspire.Hosting.Kind/KindConfigAnnotation.cs Adds annotations for full node image override and node mounts.
src/CommunityToolkit.Aspire.Hosting.Kind/KindClusterResourceBuilderExtensions.cs Adds WithNodeImage and WithNodeMount exported APIs.
src/CommunityToolkit.Aspire.Hosting.Kind/KindClusterLifecycleHook.cs Registers additional shutdown/process-exit handlers for best-effort cleanup.
src/CommunityToolkit.Aspire.Hosting.Kind/K8sManifestResource.cs Introduces new resource type representing an applied manifest.
src/CommunityToolkit.Aspire.Hosting.Kind/IProcessRunner.cs Extends process runner API to support passing stdin content.
src/CommunityToolkit.Aspire.Hosting.Kind/HelmManager.cs Adds Helm CRD-race detection, CRD snapshotting, and retry with backoff.
src/CommunityToolkit.Aspire.Hosting.Kind/DefaultProcessRunner.cs Adds stdin piping and kubeconfig path redaction in debug logging.
src/CommunityToolkit.Aspire.Hosting.Kind/CrdWaitBehavior.cs Adds exported enum controlling CRD wait failure behavior.
src/CommunityToolkit.Aspire.Hosting.Kind/api/CommunityToolkit.Aspire.Hosting.Kind.cs Updates API baseline to reflect new exported APIs/types.
examples/kind/CommunityToolkit.Aspire.Hosting.Kind.AppHost/Program.cs Demonstrates node mounts, manifests apply, SSA, and waiting on manifest resources.
examples/kind/CommunityToolkit.Aspire.Hosting.Kind.AppHost/manifests/extra-config.yaml Adds example multi-document Kubernetes manifest used by the sample.
examples/kind/CommunityToolkit.Aspire.Hosting.Kind.AppHost/CommunityToolkit.Aspire.Hosting.Kind.AppHost.csproj Copies manifest files to output for the sample AppHost.
Comments suppressed due to low confidence (2)

src/CommunityToolkit.Aspire.Hosting.Kind/KubectlManager.cs:268

  • CreateClusterInfoArguments should reject empty kubeconfigPath values up front (not just null) to avoid producing a kubectl command that will certainly fail.
    internal static IReadOnlyList<string> CreateClusterInfoArguments(string kubeconfigPath)
    {
        ArgumentNullException.ThrowIfNull(kubeconfigPath);

src/CommunityToolkit.Aspire.Hosting.Kind/KubectlManager.cs:279

  • CreateGetCrdsArguments should reject empty kubeconfigPath values up front (not just null) to avoid producing a kubectl command that will certainly fail.
    internal static IReadOnlyList<string> CreateGetCrdsArguments(string kubeconfigPath)
    {
        ArgumentNullException.ThrowIfNull(kubeconfigPath);

Comment on lines +142 to +151
internal static TimeSpan ComputeRetryBackoff(TimeSpan initialBackoff, int failureCount)
{
ArgumentOutOfRangeException.ThrowIfLessThan(failureCount, 1);

var multiplier = 1 << (failureCount - 1);
var scaledTicks = initialBackoff.Ticks * multiplier;
return scaledTicks >= TimeSpan.MaxValue.Ticks
? TimeSpan.MaxValue
: TimeSpan.FromTicks(scaledTicks);
}
Comment on lines +81 to +85
if (result.ExitCode != 0)
{
throw new InvalidOperationException(
$"Failed to apply manifest '{resource.ManifestPath}' to cluster '{resource.Parent.Name}': {result.Error}");
}
Comment on lines +246 to +248
ArgumentNullException.ThrowIfNull(crdNames);
ArgumentNullException.ThrowIfNull(kubeconfigPath);

Comment on lines +188 to +190
ArgumentNullException.ThrowIfNull(builder);
ArgumentException.ThrowIfNullOrWhiteSpace(hostPath);
ArgumentException.ThrowIfNullOrWhiteSpace(containerPath);
Comment on lines +40 to +43
ArgumentNullException.ThrowIfNull(builder);
ArgumentNullException.ThrowIfNull(name);
ArgumentNullException.ThrowIfNull(manifestPath);

Comment on lines +112 to +115
var processRunner = e.Services.GetRequiredService<IProcessRunner>();
var kubectlManager = CreateKubectlManager(processRunner, resource);
await kubectlManager.ApplyAsync(resource, logger, ct);

Comment on lines +118 to +127
private void RunCleanupSynchronously()
{
try
{
EnsureCleanupStarted().GetAwaiter().GetResult();
}
catch
{
}
}
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants